[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:29:13 dist-test-slave-wl2h krb5kdc[24999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:29:13 dist-test-slave-wl2h krb5kdc[24999](info): set up 2 sockets
Aug 07 11:29:13 dist-test-slave-wl2h krb5kdc[24999](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 11:29:14.542290 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:29:14 dist-test-slave-wl2h krb5kdc[24999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:29:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:29:14.761386 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44157
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:44157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:15.166792 25014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:15.167244 25014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:15.167412 25014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:15.175140 25014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:15.175295 25014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:15.175407 25014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:15.175519 25014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:15.175612 25014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:29:15.186043 25014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:44157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:15.187680 25014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:15.189095 25014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:15.201889 25019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:15.201844 25020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:15.201933 25023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:15.202966 25014 server_base.cc:1034] running on GCE node
I20260807 11:29:15.203866 25014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:15.205340 25014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:15.206604 25014 hybrid_clock.cc:648] HybridClock initialized: now 1786102155206571 us; error 114 us; skew 500 ppm
I20260807 11:29:15.206979 25014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:15 dist-test-slave-wl2h krb5kdc[24999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102155, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:15.210657 25014 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:29:15.212476 25014 webserver.cc:460] Webserver started at http://127.24.103.126:42839/ using document root <none> and password file <none>
I20260807 11:29:15.213198 25014 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:15.213356 25014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:15.213677 25014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:15.217235 25014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "c8d08824323c4b23ada0480d16a754ba"
format_stamp: "Formatted at 2026-08-07 11:29:15 on dist-test-slave-wl2h"
server_key: "6335a2c4636665879ffc0245ff560c5b"
server_key_iv: "e36d676631c2aaf0fd5baf77b2593946"
server_key_version: "encryptionkey@0"
I20260807 11:29:15.218025 25014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "c8d08824323c4b23ada0480d16a754ba"
format_stamp: "Formatted at 2026-08-07 11:29:15 on dist-test-slave-wl2h"
server_key: "6335a2c4636665879ffc0245ff560c5b"
server_key_iv: "e36d676631c2aaf0fd5baf77b2593946"
server_key_version: "encryptionkey@0"
I20260807 11:29:15.223702 25014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 11:29:15.227643 25030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:15.229033 25014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:29:15.229275 25014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "c8d08824323c4b23ada0480d16a754ba"
format_stamp: "Formatted at 2026-08-07 11:29:15 on dist-test-slave-wl2h"
server_key: "6335a2c4636665879ffc0245ff560c5b"
server_key_iv: "e36d676631c2aaf0fd5baf77b2593946"
server_key_version: "encryptionkey@0"
I20260807 11:29:15.229490 25014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:15.252882 25014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:15.257632 25014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:15.257965 25014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:15.279256 25014 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:44157
I20260807 11:29:15.279381 25091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:44157 every 8 connection(s)
I20260807 11:29:15.280699 25014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:15.284825 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:15.285730 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 25014
I20260807 11:29:15.286095 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:15.288968 24989 external_mini_cluster.cc:1428] Setting key 491f88ee494c4fadb5d6286fd57c2671
I20260807 11:29:15.298704 25092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba: Bootstrap starting.
I20260807 11:29:15.302001 25092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:15.303133 25092 log.cc:824] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:15.307219 25092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba: No bootstrap required, opened a new log
I20260807 11:29:15.316561 25092 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d08824323c4b23ada0480d16a754ba" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44157 } }
I20260807 11:29:15.317121 25092 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:15.317242 25092 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d08824323c4b23ada0480d16a754ba, State: Initialized, Role: FOLLOWER
I20260807 11:29:15.317898 25092 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d08824323c4b23ada0480d16a754ba" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44157 } }
I20260807 11:29:15.318123 25092 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:15.318301 25092 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:15.318501 25092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:15.321370 25092 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d08824323c4b23ada0480d16a754ba" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44157 } }
I20260807 11:29:15.322031 25092 leader_election.cc:302] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8d08824323c4b23ada0480d16a754ba; no voters:
Aug 07 11:29:15 dist-test-slave-wl2h krb5kdc[24999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:15.322794 25092 leader_election.cc:288] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:15.322978 25098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:15.324563 25098 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [term 1 LEADER]: Becoming Leader. State: Replica: c8d08824323c4b23ada0480d16a754ba, State: Running, Role: LEADER
I20260807 11:29:15.325227 25098 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d08824323c4b23ada0480d16a754ba" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44157 } }
I20260807 11:29:15.325621 25092 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:15.331321 25099 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8d08824323c4b23ada0480d16a754ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d08824323c4b23ada0480d16a754ba" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44157 } } }
I20260807 11:29:15.331121 25100 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8d08824323c4b23ada0480d16a754ba. Latest consensus state: current_term: 1 leader_uuid: "c8d08824323c4b23ada0480d16a754ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d08824323c4b23ada0480d16a754ba" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44157 } } }
I20260807 11:29:15.348464 25099 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:15.348577 25100 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:15.359333 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:15.294199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.24.103.126:44157)
0807 11:29:15.295248 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:15.295266 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:15.296609 (+ 1343us) server_negotiation.cc:407] Connection header received
0807 11:29:15.307081 (+ 10472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:15.307112 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:15.307669 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:15.308202 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:15.311252 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:15.313447 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:15.315901 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:15.316975 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:15.317213 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:15.324939 (+ 7726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:15.324955 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:15.324978 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:15.325045 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:15.331630 (+ 6585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:15.347084 (+ 15454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:15.347097 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:15.347108 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:15.348734 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:15.351355 (+ 2621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:15.351361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:15.351365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:15.352708 (+ 1343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:15.353273 (+ 565us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:15.356913 (+ 3640us) server_negotiation.cc:299] Negotiation successful
0807 11:29:15.357497 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":469,"threads_started":1}
W20260807 11:29:15.367377 25116 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:29:15.367535 25116 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:29:15.367714 25107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:15.370059 25107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:15.384652 25107 catalog_manager.cc:1347] Generated new cluster ID: c073d1745930477088a8a1d0a9a3c4fe
I20260807 11:29:15.384980 25107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:15.408669 25107 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:15.410040 25107 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:15.416549 25107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8d08824323c4b23ada0480d16a754ba: Generated new TSK 0
I20260807 11:29:15.417615 25107 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:29:15.479460 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44157
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:15.832875 25121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:15.833276 25121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:15.833461 25121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:15.840982 25121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:15.841159 25121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:15.841344 25121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:29:15.851353 25121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:15.852703 25121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:15.854053 25121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:15.868697 25126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:15.868781 25129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:15.868781 25127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:15.869917 25121 server_base.cc:1034] running on GCE node
I20260807 11:29:15.870674 25121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:15.872099 25121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:15.873320 25121 hybrid_clock.cc:648] HybridClock initialized: now 1786102155873277 us; error 32 us; skew 500 ppm
I20260807 11:29:15.873673 25121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:15 dist-test-slave-wl2h krb5kdc[24999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102155, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:15.877084 25121 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:29:15.878690 25121 webserver.cc:460] Webserver started at http://127.24.103.65:42205/ using document root <none> and password file <none>
I20260807 11:29:15.879446 25121 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:15.879554 25121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:15.879848 25121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:15.882869 25121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "1442e40919de4a6ebef90d8cb8e5ae57"
format_stamp: "Formatted at 2026-08-07 11:29:15 on dist-test-slave-wl2h"
server_key: "d50a7758a89f14b673e78a32855a7839"
server_key_iv: "18da6ad9db882acb61dd27b4eb5a0ab9"
server_key_version: "encryptionkey@0"
I20260807 11:29:15.883721 25121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "1442e40919de4a6ebef90d8cb8e5ae57"
format_stamp: "Formatted at 2026-08-07 11:29:15 on dist-test-slave-wl2h"
server_key: "d50a7758a89f14b673e78a32855a7839"
server_key_iv: "18da6ad9db882acb61dd27b4eb5a0ab9"
server_key_version: "encryptionkey@0"
I20260807 11:29:15.889060 25121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:29:15.892632 25136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:15.893940 25121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:29:15.894117 25121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "1442e40919de4a6ebef90d8cb8e5ae57"
format_stamp: "Formatted at 2026-08-07 11:29:15 on dist-test-slave-wl2h"
server_key: "d50a7758a89f14b673e78a32855a7839"
server_key_iv: "18da6ad9db882acb61dd27b4eb5a0ab9"
server_key_version: "encryptionkey@0"
I20260807 11:29:15.894336 25121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:15.908978 25121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:15.913441 25121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:15.913790 25121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:15.914875 25121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:15.916404 25121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:15.916512 25121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:15.916677 25121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:15.916756 25121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:15.953594 25121 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:34967
I20260807 11:29:15.953648 25269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:34967 every 8 connection(s)
I20260807 11:29:15.955073 25121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:29:15.961658 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 25121
I20260807 11:29:15.961921 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:29:15.962628 24989 external_mini_cluster.cc:1428] Setting key ff205d7282b53e9c59cda018af705213
Aug 07 11:29:15 dist-test-slave-wl2h krb5kdc[24999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102155, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:15.979282 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:15.957738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:46623 (local address 127.24.103.126:44157)
0807 11:29:15.958245 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:15.958249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:15.959131 (+ 882us) server_negotiation.cc:407] Connection header received
0807 11:29:15.961134 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:15.961141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:15.961236 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:15.961357 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:15.963054 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:15.964006 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:15.966424 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:15.966833 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:15.966934 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:15.971727 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:15.971733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:15.971736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:15.971771 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:15.975650 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:15.976523 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:15.976532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:15.976539 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:15.976702 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:15.977132 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:15.977140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:15.977146 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:15.977579 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:15.977735 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:15.978455 (+ 720us) server_negotiation.cc:299] Negotiation successful
0807 11:29:15.978734 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":274,"threads_started":1}
I20260807 11:29:15.979988 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:15.958214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44157 (local address 127.24.103.65:46623)
0807 11:29:15.958930 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:15.958968 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:15.960867 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:15.961594 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:15.961602 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:15.962242 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:15.962862 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:15.962882 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:15.964180 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:15.964184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:15.966251 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:15.966270 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:15.967110 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:15.967120 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:15.967480 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:15.968360 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:15.968394 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:15.971471 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:15.975854 (+ 4383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:15.975862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:15.975877 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:15.976386 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:15.976820 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:15.976823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:15.976826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:15.977012 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:15.977720 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:15.977726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:15.978072 (+ 346us) client_negotiation.cc:769] Sending connection context
0807 11:29:15.978310 (+ 238us) client_negotiation.cc:240] Negotiation successful
0807 11:29:15.978721 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":272,"threads_started":1}
I20260807 11:29:15.981776 25270 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44157
I20260807 11:29:15.982149 25270 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:15.983196 25270 heartbeater.cc:511] Master 127.24.103.126:44157 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260807 11:29:15.985775 25052 ts_manager.cc:194] Registered new tserver with Master: 1442e40919de4a6ebef90d8cb8e5ae57 (127.24.103.65:34967)
I20260807 11:29:15.987630 25052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:46623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:29:16.016376 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44157
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:16.357438 25277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:16.357736 25277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:16.357908 25277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:16.365878 25277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:16.366015 25277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:16.366240 25277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:29:16.376994 25277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:16.378437 25277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:16.379895 25277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:16.393746 25288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:16.393829 25284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:16.393672 25283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:16.398126 25277 server_base.cc:1034] running on GCE node
I20260807 11:29:16.398936 25277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:16.400025 25277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:16.401337 25277 hybrid_clock.cc:648] HybridClock initialized: now 1786102156401283 us; error 71 us; skew 500 ppm
I20260807 11:29:16.401697 25277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:16 dist-test-slave-wl2h krb5kdc[24999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102156, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:16.405002 25277 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:29:16.406548 25277 webserver.cc:460] Webserver started at http://127.24.103.66:42685/ using document root <none> and password file <none>
I20260807 11:29:16.407359 25277 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:16.407469 25277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:16.407747 25277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:16.410808 25277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "cbc473e14fa64fa38dc85f5563b73a52"
format_stamp: "Formatted at 2026-08-07 11:29:16 on dist-test-slave-wl2h"
server_key: "46468e55d637cb5333b49011c6e71757"
server_key_iv: "bd9a3ff7e293e0f11bdb9283811d6c13"
server_key_version: "encryptionkey@0"
I20260807 11:29:16.411685 25277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "cbc473e14fa64fa38dc85f5563b73a52"
format_stamp: "Formatted at 2026-08-07 11:29:16 on dist-test-slave-wl2h"
server_key: "46468e55d637cb5333b49011c6e71757"
server_key_iv: "bd9a3ff7e293e0f11bdb9283811d6c13"
server_key_version: "encryptionkey@0"
I20260807 11:29:16.417181 25277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:29:16.420902 25296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:16.422395 25277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:29:16.422544 25277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "cbc473e14fa64fa38dc85f5563b73a52"
format_stamp: "Formatted at 2026-08-07 11:29:16 on dist-test-slave-wl2h"
server_key: "46468e55d637cb5333b49011c6e71757"
server_key_iv: "bd9a3ff7e293e0f11bdb9283811d6c13"
server_key_version: "encryptionkey@0"
I20260807 11:29:16.422735 25277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:16.459990 25277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:16.465022 25277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:16.465338 25277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:16.466446 25277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:16.468072 25277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:16.468150 25277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:16.468302 25277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:16.468354 25277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:16.508984 25277 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:37845
I20260807 11:29:16.509021 25437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:37845 every 8 connection(s)
I20260807 11:29:16.510355 25277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:29:16.519068 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 25277
I20260807 11:29:16.519356 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:29:16.520066 24989 external_mini_cluster.cc:1428] Setting key 6c6ca47ffc1de179199eba3beccd3d7d
Aug 07 11:29:16 dist-test-slave-wl2h krb5kdc[24999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102156, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:16.537114 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:16.513687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:44423 (local address 127.24.103.126:44157)
0807 11:29:16.514272 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:16.514282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:16.515321 (+ 1039us) server_negotiation.cc:407] Connection header received
0807 11:29:16.517302 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:16.517311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:16.517437 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:16.517588 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:16.519165 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:16.520004 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:16.521969 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:16.522863 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:16.523071 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:16.528324 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:16.528335 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:16.528341 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:16.528395 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:16.532576 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:16.533649 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:16.533658 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:16.533665 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:16.533886 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:16.534521 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:16.534529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:16.534534 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:16.535038 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:16.535236 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:16.536270 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0807 11:29:16.536562 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":261,"threads_started":1}
I20260807 11:29:16.538372 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:16.514270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44157 (local address 127.24.103.66:44423)
0807 11:29:16.515144 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:16.515197 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:16.517018 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:16.517796 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:16.517804 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:16.518351 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:16.518921 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:16.518938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:16.520190 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:16.520197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:16.521815 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:16.521830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:16.523172 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:16.523182 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:16.523495 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:16.524374 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:16.524417 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:16.527980 (+ 3563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:16.532762 (+ 4782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:16.532776 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:16.532799 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:16.533504 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:16.534018 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:16.534026 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:16.534032 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:16.534327 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:16.535201 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:16.535212 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:16.535752 (+ 540us) client_negotiation.cc:769] Sending connection context
0807 11:29:16.536125 (+ 373us) client_negotiation.cc:240] Negotiation successful
0807 11:29:16.536750 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":380,"threads_started":1}
I20260807 11:29:16.540165 25438 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44157
I20260807 11:29:16.540621 25438 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:16.541714 25438 heartbeater.cc:511] Master 127.24.103.126:44157 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260807 11:29:16.543762 25052 ts_manager.cc:194] Registered new tserver with Master: cbc473e14fa64fa38dc85f5563b73a52 (127.24.103.66:37845)
I20260807 11:29:16.544770 25052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:44423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:29:16.571588 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44157
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:16.921185 25444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:16.921473 25444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:16.921648 25444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:16.929320 25444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:16.929448 25444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:16.929641 25444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:29:16.941174 25444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:16.942739 25444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:16.944232 25444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:16.959570 25453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:16.959756 25452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:16.960220 25455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:16.961256 25444 server_base.cc:1034] running on GCE node
I20260807 11:29:16.961947 25444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:16.962931 25444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:16.964164 25444 hybrid_clock.cc:648] HybridClock initialized: now 1786102156964103 us; error 38 us; skew 500 ppm
I20260807 11:29:16.964521 25444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:16 dist-test-slave-wl2h krb5kdc[24999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102156, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:16.973356 25444 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:29:16.975525 25444 webserver.cc:460] Webserver started at http://127.24.103.67:35405/ using document root <none> and password file <none>
I20260807 11:29:16.976336 25444 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:16.976516 25444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:16.976872 25444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:16.980284 25444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "b0a7c4a364fd418ea58681e7bbd082ed"
format_stamp: "Formatted at 2026-08-07 11:29:16 on dist-test-slave-wl2h"
server_key: "b20b66e59eac8f4bbfb496e4b63dc0cd"
server_key_iv: "00bb9cf2c8bcc78bb87475ba95cddcea"
server_key_version: "encryptionkey@0"
I20260807 11:29:16.981174 25444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "b0a7c4a364fd418ea58681e7bbd082ed"
format_stamp: "Formatted at 2026-08-07 11:29:16 on dist-test-slave-wl2h"
server_key: "b20b66e59eac8f4bbfb496e4b63dc0cd"
server_key_iv: "00bb9cf2c8bcc78bb87475ba95cddcea"
server_key_version: "encryptionkey@0"
I20260807 11:29:16.987398 25444 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 11:29:16.990850 25270 heartbeater.cc:503] Master 127.24.103.126:44157 was elected leader, sending a full tablet report...
I20260807 11:29:16.991358 25463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:16.992997 25444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:29:16.993191 25444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "b0a7c4a364fd418ea58681e7bbd082ed"
format_stamp: "Formatted at 2026-08-07 11:29:16 on dist-test-slave-wl2h"
server_key: "b20b66e59eac8f4bbfb496e4b63dc0cd"
server_key_iv: "00bb9cf2c8bcc78bb87475ba95cddcea"
server_key_version: "encryptionkey@0"
I20260807 11:29:16.993409 25444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:17.013120 25444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:17.018283 25444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:17.018627 25444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:17.019837 25444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:17.021456 25444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:17.021596 25444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:17.021759 25444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:17.021835 25444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:17.059753 25444 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:43823
I20260807 11:29:17.059777 25604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:43823 every 8 connection(s)
I20260807 11:29:17.061185 25444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:29:17.064347 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 25444
I20260807 11:29:17.064603 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:29:17.065182 24989 external_mini_cluster.cc:1428] Setting key 98214ccfb486a561959ebcce9c17eae7
Aug 07 11:29:17 dist-test-slave-wl2h krb5kdc[24999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102156, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:17.084409 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:17.064041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:54697 (local address 127.24.103.126:44157)
0807 11:29:17.064711 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:17.064719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:17.065880 (+ 1161us) server_negotiation.cc:407] Connection header received
0807 11:29:17.067954 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:17.067959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:17.068039 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:17.068163 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:17.070072 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.070912 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.072618 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.073064 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.073164 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:17.077194 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:17.077200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:17.077203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:17.077238 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:17.080523 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.081585 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.081590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.081593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.081817 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.082534 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.082539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.082541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.082850 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:17.082987 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:17.083917 (+ 930us) server_negotiation.cc:299] Negotiation successful
0807 11:29:17.084100 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":347,"threads_started":1}
I20260807 11:29:17.085390 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:17.064585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44157 (local address 127.24.103.67:54697)
0807 11:29:17.065652 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:17.065705 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:17.067703 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:17.068423 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:17.068432 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:17.069335 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:17.069903 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:17.069920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.071141 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.071145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:17.072474 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:17.072487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.073364 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.073371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:17.073709 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:17.074506 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:17.074546 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:17.076971 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:17.080731 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:17.080744 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:17.080762 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:17.081433 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:17.082057 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:17.082063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:17.082070 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:17.082360 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:17.082988 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:17.082994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:17.083479 (+ 485us) client_negotiation.cc:769] Sending connection context
0807 11:29:17.083748 (+ 269us) client_negotiation.cc:240] Negotiation successful
0807 11:29:17.084196 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":417,"threads_started":1}
I20260807 11:29:17.087133 25606 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44157
I20260807 11:29:17.087476 25606 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:17.088263 25606 heartbeater.cc:511] Master 127.24.103.126:44157 requested a full tablet report, sending...
I20260807 11:29:17.089957 25052 ts_manager.cc:194] Registered new tserver with Master: b0a7c4a364fd418ea58681e7bbd082ed (127.24.103.67:43823)
I20260807 11:29:17.090862 25052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:54697
I20260807 11:29:17.095199 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:29:17 dist-test-slave-wl2h krb5kdc[24999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102157, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 11:29:17 dist-test-slave-wl2h krb5kdc[24999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102157, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:29:17.151698 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:17.134273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.24.103.65:34967)
0807 11:29:17.134975 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:17.134983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:17.135059 (+ 76us) server_negotiation.cc:407] Connection header received
0807 11:29:17.135266 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:17.135273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:17.135493 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:17.135630 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:17.137275 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.138310 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.139647 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.140239 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.140344 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:17.144653 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:17.144667 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:17.144683 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:17.144725 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:17.148303 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.149142 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.149188 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.149190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.149341 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.149876 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.149880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.149883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.150334 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:17.150608 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:17.151110 (+ 502us) server_negotiation.cc:299] Negotiation successful
0807 11:29:17.151378 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":359,"threads_started":1}
W20260807 11:29:17.153321 25169 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36748
Aug 07 11:29:17 dist-test-slave-wl2h krb5kdc[24999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 11:29:17 dist-test-slave-wl2h krb5kdc[24999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:29:17.208842 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:17.191812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.24.103.65:34967)
0807 11:29:17.192087 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:17.192096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:17.192451 (+ 355us) server_negotiation.cc:407] Connection header received
0807 11:29:17.192598 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:17.192602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:17.192688 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:17.192778 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:17.194230 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.194948 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.196348 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.196795 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.196909 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:17.202047 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:17.202053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:17.202057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:17.202098 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:17.205702 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.206509 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.206514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.206517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.206668 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.207362 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.207371 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.207377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.207702 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:17.207904 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:17.208377 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 11:29:17.208577 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 11:29:17.212189 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 25121
I20260807 11:29:17.227514 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:17.213418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.24.103.65:34967)
0807 11:29:17.213688 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:17.213698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:17.214030 (+ 332us) server_negotiation.cc:407] Connection header received
0807 11:29:17.214241 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:17.214246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:17.214319 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:17.214431 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:17.215841 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.216571 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.217923 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.218383 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.218533 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:17.221415 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:17.221421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:17.221425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:17.221465 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:17.224568 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.225341 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.225348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.225351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.225515 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.225942 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.225948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.225951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.226246 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:17.226519 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:17.226859 (+ 340us) server_negotiation.cc:299] Negotiation successful
0807 11:29:17.227142 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:29:17.228386 25265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:17.326241 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 25121
I20260807 11:29:17.349014 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 25277
Aug 07 11:29:17 dist-test-slave-wl2h krb5kdc[24999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:29:17.367172 25624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:17.350142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.24.103.66:37845)
0807 11:29:17.350728 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:17.350735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:17.350761 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:29:17.350830 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:17.350837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:17.351106 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:17.351309 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:17.352598 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.353840 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.355151 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.355719 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.355863 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:17.360424 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:17.360438 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:17.360454 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:17.360494 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:17.364033 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.364764 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.364780 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.364782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.364916 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.365350 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.365355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.365360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.365737 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:17.366069 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:17.366412 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 11:29:17.366709 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":285,"threads_started":1}
I20260807 11:29:17.368574 25432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:17.457443 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 25277
I20260807 11:29:17.476994 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 25444
Aug 07 11:29:17 dist-test-slave-wl2h krb5kdc[24999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:29:17.494419 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:17.478283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54074 (local address 127.24.103.67:43823)
0807 11:29:17.478984 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:17.478991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:17.479102 (+ 111us) server_negotiation.cc:407] Connection header received
0807 11:29:17.479175 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:17.479181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:17.479399 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:17.479534 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:17.480810 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.481876 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.483351 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:17.483955 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:17.484068 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:17.488412 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:17.488424 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:17.488439 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:17.488481 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:17.491624 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.492351 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.492358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.492361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.492516 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:17.492889 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:17.492893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:17.492895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:17.493302 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:17.493555 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:17.493886 (+ 331us) server_negotiation.cc:299] Negotiation successful
0807 11:29:17.494115 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":345,"threads_started":1}
I20260807 11:29:17.495985 25598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:17.580884 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 25444
I20260807 11:29:17.600162 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 25014
I20260807 11:29:17.601744 25087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:17.709352 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 25014
2026-08-07T11:29:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4454 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:29:17 dist-test-slave-wl2h krb5kdc[25632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:29:17 dist-test-slave-wl2h krb5kdc[25632](info): set up 2 sockets
Aug 07 11:29:17 dist-test-slave-wl2h krb5kdc[25632](info): commencing operation
krb5kdc: starting...
W20260807 11:29:18.912042 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:29:19 dist-test-slave-wl2h krb5kdc[25632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:29:19.135625 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44815
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:44815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:19.469039 25646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:19.469389 25646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:19.469551 25646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:19.477065 25646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:19.477223 25646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:19.477340 25646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:19.477438 25646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:19.477525 25646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:29:19.488025 25646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:44815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:19.489459 25646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:19.490844 25646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:19.502878 25655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:19.502961 25651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:19.502990 25652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:19.504464 25646 server_base.cc:1034] running on GCE node
I20260807 11:29:19.505249 25646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:19.506719 25646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:19.507958 25646 hybrid_clock.cc:648] HybridClock initialized: now 1786102159507919 us; error 82 us; skew 500 ppm
I20260807 11:29:19.508318 25646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:19 dist-test-slave-wl2h krb5kdc[25632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102159, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:19.511428 25646 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:29:19.512985 25646 webserver.cc:460] Webserver started at http://127.24.103.126:41777/ using document root <none> and password file <none>
I20260807 11:29:19.513695 25646 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:19.513851 25646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:19.514178 25646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:19.517194 25646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "6815b52ceb1748c7b349aeb62b5f00fb"
format_stamp: "Formatted at 2026-08-07 11:29:19 on dist-test-slave-wl2h"
server_key: "2fdd0a748f700a1f9bee6a9225f3aa5a"
server_key_iv: "e4b1dfc1f50985a9f7a0340b7e8e37e3"
server_key_version: "encryptionkey@0"
I20260807 11:29:19.517971 25646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "6815b52ceb1748c7b349aeb62b5f00fb"
format_stamp: "Formatted at 2026-08-07 11:29:19 on dist-test-slave-wl2h"
server_key: "2fdd0a748f700a1f9bee6a9225f3aa5a"
server_key_iv: "e4b1dfc1f50985a9f7a0340b7e8e37e3"
server_key_version: "encryptionkey@0"
I20260807 11:29:19.523213 25646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260807 11:29:19.526654 25663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:19.528009 25646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:19.528195 25646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "6815b52ceb1748c7b349aeb62b5f00fb"
format_stamp: "Formatted at 2026-08-07 11:29:19 on dist-test-slave-wl2h"
server_key: "2fdd0a748f700a1f9bee6a9225f3aa5a"
server_key_iv: "e4b1dfc1f50985a9f7a0340b7e8e37e3"
server_key_version: "encryptionkey@0"
I20260807 11:29:19.528410 25646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:19.556985 25646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:19.561285 25646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:19.561595 25646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:19.581224 25646 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:44815
I20260807 11:29:19.581255 25722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:44815 every 8 connection(s)
I20260807 11:29:19.582657 25646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:19.586421 25723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:19.586977 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 25646
I20260807 11:29:19.587325 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:19.588214 24989 external_mini_cluster.cc:1428] Setting key 05f7205ea55a2035b1c440b80fd98070
I20260807 11:29:19.599188 25723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb: Bootstrap starting.
Aug 07 11:29:19 dist-test-slave-wl2h krb5kdc[25632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:19.602561 25723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:19.603853 25723 log.cc:824] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:19.606905 25723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb: No bootstrap required, opened a new log
I20260807 11:29:19.611326 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:19.590476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.24.103.126:44815)
0807 11:29:19.591333 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:19.591345 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:19.591388 (+ 43us) server_negotiation.cc:407] Connection header received
0807 11:29:19.593069 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:19.593090 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:19.593482 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:19.593871 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:19.595463 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:19.596756 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:19.598035 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:19.598714 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:19.598919 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:19.603456 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:19.603464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:19.603480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:19.603515 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:19.606987 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:19.607762 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:19.607771 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:19.607778 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:19.607961 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:19.608346 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:19.608352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:19.608355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:19.608893 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:19.609268 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:19.609806 (+ 538us) server_negotiation.cc:299] Negotiation successful
0807 11:29:19.610235 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":321,"threads_started":1}
I20260807 11:29:19.615227 25723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b52ceb1748c7b349aeb62b5f00fb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44815 } }
I20260807 11:29:19.615574 25723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:19.615701 25723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815b52ceb1748c7b349aeb62b5f00fb, State: Initialized, Role: FOLLOWER
I20260807 11:29:19.616246 25723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b52ceb1748c7b349aeb62b5f00fb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44815 } }
I20260807 11:29:19.616451 25723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:19.616602 25723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:19.616770 25723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:19.619463 25723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b52ceb1748c7b349aeb62b5f00fb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44815 } }
I20260807 11:29:19.620147 25723 leader_election.cc:302] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6815b52ceb1748c7b349aeb62b5f00fb; no voters:
I20260807 11:29:19.620844 25723 leader_election.cc:288] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:19.620978 25731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:19.621737 25731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [term 1 LEADER]: Becoming Leader. State: Replica: 6815b52ceb1748c7b349aeb62b5f00fb, State: Running, Role: LEADER
I20260807 11:29:19.622406 25731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b52ceb1748c7b349aeb62b5f00fb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44815 } }
I20260807 11:29:19.623302 25723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:19.625977 25733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6815b52ceb1748c7b349aeb62b5f00fb. Latest consensus state: current_term: 1 leader_uuid: "6815b52ceb1748c7b349aeb62b5f00fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b52ceb1748c7b349aeb62b5f00fb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44815 } } }
I20260807 11:29:19.626375 25733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:19.626156 25732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6815b52ceb1748c7b349aeb62b5f00fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b52ceb1748c7b349aeb62b5f00fb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44815 } } }
I20260807 11:29:19.626597 25732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:19.628377 25738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:19.633606 25738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:19.641901 25738 catalog_manager.cc:1347] Generated new cluster ID: ae8070ad43b34252ac33871c5efb777a
I20260807 11:29:19.642019 25738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:19.659653 25738 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:19.661744 25738 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:19.668751 25738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6815b52ceb1748c7b349aeb62b5f00fb: Generated new TSK 0
I20260807 11:29:19.669636 25738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:29:19.731297 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44815
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:20.081820 25754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:20.082124 25754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:20.082226 25754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:20.089604 25754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:20.089735 25754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:20.089854 25754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:29:20.100909 25754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:20.102334 25754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:20.103722 25754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:20.117002 25761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:20.117107 25760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:20.117038 25764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:20.118428 25754 server_base.cc:1034] running on GCE node
I20260807 11:29:20.119114 25754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:20.120067 25754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:20.121308 25754 hybrid_clock.cc:648] HybridClock initialized: now 1786102160121257 us; error 57 us; skew 500 ppm
I20260807 11:29:20.121659 25754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:20 dist-test-slave-wl2h krb5kdc[25632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:20.125017 25754 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:29:20.126482 25754 webserver.cc:460] Webserver started at http://127.24.103.65:41417/ using document root <none> and password file <none>
I20260807 11:29:20.127280 25754 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:20.127434 25754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:20.127780 25754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:20.130769 25754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "32d078ed11c9454e9ee09278f8af64bd"
format_stamp: "Formatted at 2026-08-07 11:29:20 on dist-test-slave-wl2h"
server_key: "1ff8814b9d0dbe82ff8738602a2c5a4f"
server_key_iv: "cb99c92ae9b4fb2ddde6e349a4b34fd2"
server_key_version: "encryptionkey@0"
I20260807 11:29:20.131636 25754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "32d078ed11c9454e9ee09278f8af64bd"
format_stamp: "Formatted at 2026-08-07 11:29:20 on dist-test-slave-wl2h"
server_key: "1ff8814b9d0dbe82ff8738602a2c5a4f"
server_key_iv: "cb99c92ae9b4fb2ddde6e349a4b34fd2"
server_key_version: "encryptionkey@0"
I20260807 11:29:20.136890 25754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:29:20.140281 25773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:20.141548 25754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:29:20.141729 25754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "32d078ed11c9454e9ee09278f8af64bd"
format_stamp: "Formatted at 2026-08-07 11:29:20 on dist-test-slave-wl2h"
server_key: "1ff8814b9d0dbe82ff8738602a2c5a4f"
server_key_iv: "cb99c92ae9b4fb2ddde6e349a4b34fd2"
server_key_version: "encryptionkey@0"
I20260807 11:29:20.141937 25754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:20.173678 25754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:20.178346 25754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:20.178633 25754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:20.179658 25754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:20.181349 25754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:20.181429 25754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:20.181527 25754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:20.181583 25754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:20.218257 25754 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:42045
I20260807 11:29:20.218305 25900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:42045 every 8 connection(s)
I20260807 11:29:20.219700 25754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:29:20.224895 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 25754
I20260807 11:29:20.225179 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:29:20.225823 24989 external_mini_cluster.cc:1428] Setting key 35d2ab61b72794a8d5ad124a00067065
Aug 07 11:29:20 dist-test-slave-wl2h krb5kdc[25632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:20.241693 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:20.222449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:35291 (local address 127.24.103.126:44815)
0807 11:29:20.222968 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:20.222972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:20.223932 (+ 960us) server_negotiation.cc:407] Connection header received
0807 11:29:20.226153 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:20.226159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:20.226257 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:20.226383 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:20.228334 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:20.229065 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:20.230658 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:20.231135 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:20.231277 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:20.235530 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:20.235535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:20.235539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:20.235573 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:20.238299 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:20.239279 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:20.239283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:20.239285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:20.239407 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:20.239921 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:20.239924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:20.239926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:20.240226 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:20.240356 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:20.241268 (+ 912us) server_negotiation.cc:299] Negotiation successful
0807 11:29:20.241439 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260807 11:29:20.243425 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:20.222927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44815 (local address 127.24.103.65:35291)
0807 11:29:20.223769 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:20.223807 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:20.225834 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:20.226639 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:20.226652 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:20.227352 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:20.228157 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:20.228177 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:20.229245 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:20.229250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:20.230514 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:20.230526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:20.231388 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:20.231397 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:20.231653 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:20.232438 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:20.232460 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:20.235281 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:20.238461 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:20.238473 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:20.238491 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:20.239141 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:20.239523 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:20.239531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:20.239537 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:20.239798 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:20.240386 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:20.240397 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:20.240851 (+ 454us) client_negotiation.cc:769] Sending connection context
0807 11:29:20.241154 (+ 303us) client_negotiation.cc:240] Negotiation successful
0807 11:29:20.241757 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":326,"threads_started":1}
I20260807 11:29:20.245317 25901 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44815
I20260807 11:29:20.245685 25901 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:20.246584 25901 heartbeater.cc:511] Master 127.24.103.126:44815 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260807 11:29:20.249011 25682 ts_manager.cc:194] Registered new tserver with Master: 32d078ed11c9454e9ee09278f8af64bd (127.24.103.65:42045)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
I20260807 11:29:20.250624 25682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:35291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:29:20.279889 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44815
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:20.618492 25908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:20.618777 25908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:20.618873 25908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:20.626380 25908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:20.626507 25908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:20.626632 25908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:29:20.636830 25908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:20.638154 25908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:20.639600 25908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:20.653743 25913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:20.653780 25917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:20.653805 25914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:20.655097 25908 server_base.cc:1034] running on GCE node
I20260807 11:29:20.655679 25908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:20.656633 25908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:20.657838 25908 hybrid_clock.cc:648] HybridClock initialized: now 1786102160657782 us; error 58 us; skew 500 ppm
I20260807 11:29:20.658164 25908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:20 dist-test-slave-wl2h krb5kdc[25632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:20.661398 25908 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:29:20.662995 25908 webserver.cc:460] Webserver started at http://127.24.103.66:34339/ using document root <none> and password file <none>
I20260807 11:29:20.663787 25908 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:20.663944 25908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:20.664270 25908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:20.667358 25908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "f84cff3210e74de89c162728314cc453"
format_stamp: "Formatted at 2026-08-07 11:29:20 on dist-test-slave-wl2h"
server_key: "2220e70cce4369362937e1a156d049d0"
server_key_iv: "31ac773b5089bd4dfb061715b2d6b9f5"
server_key_version: "encryptionkey@0"
I20260807 11:29:20.668206 25908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "f84cff3210e74de89c162728314cc453"
format_stamp: "Formatted at 2026-08-07 11:29:20 on dist-test-slave-wl2h"
server_key: "2220e70cce4369362937e1a156d049d0"
server_key_iv: "31ac773b5089bd4dfb061715b2d6b9f5"
server_key_version: "encryptionkey@0"
I20260807 11:29:20.673959 25908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 11:29:20.678013 25925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:20.679523 25908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 11:29:20.679728 25908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "f84cff3210e74de89c162728314cc453"
format_stamp: "Formatted at 2026-08-07 11:29:20 on dist-test-slave-wl2h"
server_key: "2220e70cce4369362937e1a156d049d0"
server_key_iv: "31ac773b5089bd4dfb061715b2d6b9f5"
server_key_version: "encryptionkey@0"
I20260807 11:29:20.679945 25908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:20.718801 25908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:20.723219 25908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:20.723585 25908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:20.724732 25908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:20.726116 25908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:20.726223 25908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:20.726389 25908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:20.726473 25908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:20.763299 25908 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:33163
I20260807 11:29:20.763427 26055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:33163 every 8 connection(s)
I20260807 11:29:20.764695 25908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:29:20.773205 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 25908
I20260807 11:29:20.773420 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:29:20.773994 24989 external_mini_cluster.cc:1428] Setting key 080acd26e469431c031dcb8b7cfa63fa
Aug 07 11:29:20 dist-test-slave-wl2h krb5kdc[25632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:20.789049 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:20.767633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:44315 (local address 127.24.103.126:44815)
0807 11:29:20.768150 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:20.768157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:20.769281 (+ 1124us) server_negotiation.cc:407] Connection header received
0807 11:29:20.771173 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:20.771178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:20.771264 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:20.771378 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:20.772919 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:20.774074 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:20.776045 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:20.776680 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:20.776829 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:20.780668 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:20.780677 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:20.780685 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:20.780735 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:20.785068 (+ 4333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:20.786215 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:20.786222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:20.786225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:20.786436 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:20.786993 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:20.786996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:20.786998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:20.787391 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:20.787525 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:20.788537 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0807 11:29:20.788752 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":238,"threads_started":1}
I20260807 11:29:20.790694 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:20.768367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44815 (local address 127.24.103.66:44315)
0807 11:29:20.769111 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:20.769149 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:20.770850 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:20.771586 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:20.771594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:20.772155 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:20.772711 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:20.772739 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:20.774342 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:20.774347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:20.775899 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:20.775918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:20.776907 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:20.776912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:20.777120 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:20.777860 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:20.777884 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:20.780405 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:20.785246 (+ 4841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:20.785260 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:20.785282 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:20.786027 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:20.786586 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:20.786594 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:20.786600 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:20.786866 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:20.787546 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:20.787558 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:20.788055 (+ 497us) client_negotiation.cc:769] Sending connection context
0807 11:29:20.788424 (+ 369us) client_negotiation.cc:240] Negotiation successful
0807 11:29:20.789044 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":314,"threads_started":1}
I20260807 11:29:20.792325 26056 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44815
I20260807 11:29:20.792675 26056 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:20.793565 26056 heartbeater.cc:511] Master 127.24.103.126:44815 requested a full tablet report, sending...
I20260807 11:29:20.795537 25682 ts_manager.cc:194] Registered new tserver with Master: f84cff3210e74de89c162728314cc453 (127.24.103.66:33163)
I20260807 11:29:20.796492 25682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:44315
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:29:20.831265 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44815
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:21.174183 26063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:21.174489 26063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:21.174657 26063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:21.182096 26063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:21.182209 26063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:21.182425 26063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:29:21.193070 26063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:21.194483 26063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:21.195879 26063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:21.209887 26071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:21.209968 26069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:21.209986 26068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:21.210990 26063 server_base.cc:1034] running on GCE node
I20260807 11:29:21.211711 26063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:21.212720 26063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:21.214008 26063 hybrid_clock.cc:648] HybridClock initialized: now 1786102161213944 us; error 43 us; skew 500 ppm
I20260807 11:29:21.214407 26063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:21 dist-test-slave-wl2h krb5kdc[25632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102161, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:21.218001 26063 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:29:21.220676 26063 webserver.cc:460] Webserver started at http://127.24.103.67:36477/ using document root <none> and password file <none>
I20260807 11:29:21.221374 26063 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:21.221511 26063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:21.221925 26063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:21.225174 26063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "fe8a5ea2ae6b459fa2a55be41f186664"
format_stamp: "Formatted at 2026-08-07 11:29:21 on dist-test-slave-wl2h"
server_key: "aded187bd3ee141c04dccbd71f1b6195"
server_key_iv: "4a4df0540c558952bb611f33a0aee7ab"
server_key_version: "encryptionkey@0"
I20260807 11:29:21.225971 26063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "fe8a5ea2ae6b459fa2a55be41f186664"
format_stamp: "Formatted at 2026-08-07 11:29:21 on dist-test-slave-wl2h"
server_key: "aded187bd3ee141c04dccbd71f1b6195"
server_key_iv: "4a4df0540c558952bb611f33a0aee7ab"
server_key_version: "encryptionkey@0"
I20260807 11:29:21.231769 26063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 11:29:21.235730 26079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:21.237073 26063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:29:21.237262 26063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "fe8a5ea2ae6b459fa2a55be41f186664"
format_stamp: "Formatted at 2026-08-07 11:29:21 on dist-test-slave-wl2h"
server_key: "aded187bd3ee141c04dccbd71f1b6195"
server_key_iv: "4a4df0540c558952bb611f33a0aee7ab"
server_key_version: "encryptionkey@0"
I20260807 11:29:21.237474 26063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:21.253672 25901 heartbeater.cc:503] Master 127.24.103.126:44815 was elected leader, sending a full tablet report...
I20260807 11:29:21.288796 26063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:21.294116 26063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:21.294425 26063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:21.295604 26063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:21.297129 26063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:21.297214 26063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:21.297317 26063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:21.297374 26063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:21.335702 26063 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:37291
I20260807 11:29:21.335799 26216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:37291 every 8 connection(s)
I20260807 11:29:21.337052 26063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:29:21.343324 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 26063
I20260807 11:29:21.343552 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:29:21.344187 24989 external_mini_cluster.cc:1428] Setting key 87c73251f9c43e362ef6e1fd35314bbf
Aug 07 11:29:21 dist-test-slave-wl2h krb5kdc[25632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102161, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:21.360678 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.339870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:55333 (local address 127.24.103.126:44815)
0807 11:29:21.340375 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:21.340381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:21.341534 (+ 1153us) server_negotiation.cc:407] Connection header received
0807 11:29:21.343997 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:21.344002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:21.344079 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:21.344185 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:21.346541 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.347794 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.349412 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.349820 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.349920 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.353773 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:21.353778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:21.353782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:21.353819 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:21.356755 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.357702 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.357707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.357710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.357837 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.358339 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.358344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.358346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.358719 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:21.358884 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:21.360026 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0807 11:29:21.360275 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
I20260807 11:29:21.361879 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.340528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44815 (local address 127.24.103.67:55333)
0807 11:29:21.341344 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:21.341401 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:21.343646 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:21.344500 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:21.344514 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:21.345415 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:21.346266 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.346298 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.347990 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.347994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.349265 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.349278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.350066 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.350074 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.350284 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.350984 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:21.351078 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:21.353512 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:21.356920 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:21.356929 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:21.356946 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:21.357559 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:21.357943 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:21.357947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:21.357949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:21.358215 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:21.358863 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:21.358869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:21.359545 (+ 676us) client_negotiation.cc:769] Sending connection context
0807 11:29:21.359887 (+ 342us) client_negotiation.cc:240] Negotiation successful
0807 11:29:21.360570 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":396,"threads_started":1}
I20260807 11:29:21.363528 26217 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44815
I20260807 11:29:21.363844 26217 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:21.364615 26217 heartbeater.cc:511] Master 127.24.103.126:44815 requested a full tablet report, sending...
I20260807 11:29:21.366257 25682 ts_manager.cc:194] Registered new tserver with Master: fe8a5ea2ae6b459fa2a55be41f186664 (127.24.103.67:37291)
I20260807 11:29:21.367244 25682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:55333
I20260807 11:29:21.373473 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:29:21.396929 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.382053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48386 (local address 127.24.103.126:44815)
0807 11:29:21.382320 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:21.382329 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:21.382868 (+ 539us) server_negotiation.cc:407] Connection header received
0807 11:29:21.383165 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:21.383170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:21.383246 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:21.383365 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:21.385436 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.386561 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.388042 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.388464 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.388610 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.391065 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:21.391072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:21.391077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:21.391125 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:21.394028 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.394783 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.394789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.394793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.394929 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.395356 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.395364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.395367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.395715 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:21.395979 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:21.396358 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 11:29:21.396592 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:29:21.404250 25682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:29:21.411957 25682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:29:21.448549 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.426767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.24.103.66:33163)
0807 11:29:21.439652 (+ 12885us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:21.439658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:21.439683 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:29:21.439766 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:21.439771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:21.440038 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:21.440254 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:21.441105 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.442875 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.445923 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.447100 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.447267 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.447495 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:21.447875 (+ 380us) server_negotiation.cc:299] Negotiation successful
0807 11:29:21.448068 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12743,"thread_start_us":12625,"threads_started":1}
I20260807 11:29:21.449241 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.426125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:33163 (local address 127.0.0.1:48808)
0807 11:29:21.438954 (+ 12829us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:21.438972 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:21.439181 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:21.440268 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:21.440276 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:21.440308 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:21.440841 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.440855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.443054 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.443061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.445762 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.445780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.447328 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.447334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.447627 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.447676 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 11:29:21.447817 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 11:29:21.448019 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12713,"mutex_wait_us":40,"thread_start_us":14512,"threads_started":1}
I20260807 11:29:21.450443 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.425907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37291 (local address 127.0.0.1:55782)
0807 11:29:21.433162 (+ 7255us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:21.433217 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:21.433451 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:21.442439 (+ 8988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:21.442448 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:21.442500 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:21.442994 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.443008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.445220 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.445226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.447679 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.447696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.449592 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.449599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.449791 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.449825 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 11:29:21.449895 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 11:29:21.450029 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6913,"mutex_wait_us":84,"spinlock_wait_cycles":1920,"thread_start_us":10807,"threads_started":1}
I20260807 11:29:21.451458 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.432845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42045 (local address 127.0.0.1:56090)
0807 11:29:21.440987 (+ 8142us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:21.441006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:21.441119 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:21.442262 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:21.442267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:21.442294 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:21.442867 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.442884 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.444905 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.444909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.447721 (+ 2812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.447732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.450578 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.450585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.450762 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.450804 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 11:29:21.450899 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 11:29:21.451072 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7998,"mutex_wait_us":139,"thread_start_us":5848,"threads_started":1}
I20260807 11:29:21.452216 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.426971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.24.103.65:42045)
0807 11:29:21.438670 (+ 11699us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:21.438683 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:21.441095 (+ 2412us) server_negotiation.cc:407] Connection header received
0807 11:29:21.441778 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:21.441787 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:21.442041 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:21.442258 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:21.443095 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.444766 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.447901 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.449968 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.450093 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.450422 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:21.451687 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0807 11:29:21.451875 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11450,"thread_start_us":11227,"threads_started":1}
I20260807 11:29:21.453851 25981 tablet_service.cc:1467] Processing CreateTablet for tablet e255c98e5572464dbc9e6dad99ef4215 (DEFAULT_TABLE table=test-table [id=093a65e041be48c5a029ced0da3d2d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:21.455264 25981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e255c98e5572464dbc9e6dad99ef4215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:21.456410 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.426128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.24.103.67:37291)
0807 11:29:21.441722 (+ 15594us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:21.441730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:21.441762 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:29:21.441846 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:21.441853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:21.442212 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:21.443106 (+ 894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:21.443431 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.445017 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.447923 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.449350 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.455659 (+ 6309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.455864 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:21.455923 (+ 59us) server_negotiation.cc:299] Negotiation successful
0807 11:29:21.456049 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15402,"thread_start_us":16334,"threads_started":1}
I20260807 11:29:21.458730 25826 tablet_service.cc:1467] Processing CreateTablet for tablet e255c98e5572464dbc9e6dad99ef4215 (DEFAULT_TABLE table=test-table [id=093a65e041be48c5a029ced0da3d2d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:21.460340 25826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e255c98e5572464dbc9e6dad99ef4215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:21.462783 26139 tablet_service.cc:1467] Processing CreateTablet for tablet e255c98e5572464dbc9e6dad99ef4215 (DEFAULT_TABLE table=test-table [id=093a65e041be48c5a029ced0da3d2d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:21.464751 26139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e255c98e5572464dbc9e6dad99ef4215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:21.469960 26239 tablet_bootstrap.cc:492] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453: Bootstrap starting.
I20260807 11:29:21.474872 26239 tablet_bootstrap.cc:654] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:21.475660 26240 tablet_bootstrap.cc:492] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd: Bootstrap starting.
I20260807 11:29:21.476768 26239 log.cc:824] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:21.479355 26242 tablet_bootstrap.cc:492] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664: Bootstrap starting.
I20260807 11:29:21.479575 26240 tablet_bootstrap.cc:654] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:21.480414 26239 tablet_bootstrap.cc:492] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453: No bootstrap required, opened a new log
I20260807 11:29:21.480765 26239 ts_tablet_manager.cc:1397] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260807 11:29:21.481031 26240 log.cc:824] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:21.483382 26242 tablet_bootstrap.cc:654] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:21.484202 26240 tablet_bootstrap.cc:492] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd: No bootstrap required, opened a new log
I20260807 11:29:21.484491 26240 ts_tablet_manager.cc:1397] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 11:29:21.485066 26242 log.cc:824] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:21.488482 26242 tablet_bootstrap.cc:492] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664: No bootstrap required, opened a new log
I20260807 11:29:21.488780 26242 ts_tablet_manager.cc:1397] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260807 11:29:21.491954 26239 raft_consensus.cc:348] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8a5ea2ae6b459fa2a55be41f186664" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37291 } } peers { permanent_uuid: "f84cff3210e74de89c162728314cc453" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33163 } } peers { permanent_uuid: "32d078ed11c9454e9ee09278f8af64bd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42045 } }
I20260807 11:29:21.492628 26239 raft_consensus.cc:374] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:21.492745 26239 raft_consensus.cc:729] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84cff3210e74de89c162728314cc453, State: Initialized, Role: FOLLOWER
I20260807 11:29:21.492934 26240 raft_consensus.cc:348] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8a5ea2ae6b459fa2a55be41f186664" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37291 } } peers { permanent_uuid: "f84cff3210e74de89c162728314cc453" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33163 } } peers { permanent_uuid: "32d078ed11c9454e9ee09278f8af64bd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42045 } }
I20260807 11:29:21.493398 26240 raft_consensus.cc:374] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:21.493505 26240 raft_consensus.cc:729] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d078ed11c9454e9ee09278f8af64bd, State: Initialized, Role: FOLLOWER
I20260807 11:29:21.493414 26239 consensus_queue.cc:260] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8a5ea2ae6b459fa2a55be41f186664" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37291 } } peers { permanent_uuid: "f84cff3210e74de89c162728314cc453" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33163 } } peers { permanent_uuid: "32d078ed11c9454e9ee09278f8af64bd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42045 } }
I20260807 11:29:21.494153 26240 consensus_queue.cc:260] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8a5ea2ae6b459fa2a55be41f186664" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37291 } } peers { permanent_uuid: "f84cff3210e74de89c162728314cc453" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33163 } } peers { permanent_uuid: "32d078ed11c9454e9ee09278f8af64bd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42045 } }
I20260807 11:29:21.494705 26056 heartbeater.cc:503] Master 127.24.103.126:44815 was elected leader, sending a full tablet report...
I20260807 11:29:21.495956 26240 ts_tablet_manager.cc:1428] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260807 11:29:21.497038 26239 ts_tablet_manager.cc:1428] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260807 11:29:21.501031 26242 raft_consensus.cc:348] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8a5ea2ae6b459fa2a55be41f186664" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37291 } } peers { permanent_uuid: "f84cff3210e74de89c162728314cc453" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33163 } } peers { permanent_uuid: "32d078ed11c9454e9ee09278f8af64bd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42045 } }
I20260807 11:29:21.501510 26242 raft_consensus.cc:374] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:21.501652 26242 raft_consensus.cc:729] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8a5ea2ae6b459fa2a55be41f186664, State: Initialized, Role: FOLLOWER
I20260807 11:29:21.502292 26242 consensus_queue.cc:260] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8a5ea2ae6b459fa2a55be41f186664" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37291 } } peers { permanent_uuid: "f84cff3210e74de89c162728314cc453" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33163 } } peers { permanent_uuid: "32d078ed11c9454e9ee09278f8af64bd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42045 } }
I20260807 11:29:21.503489 26217 heartbeater.cc:503] Master 127.24.103.126:44815 was elected leader, sending a full tablet report...
I20260807 11:29:21.504271 26242 ts_tablet_manager.cc:1428] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260807 11:29:21.516889 26057 tablet.cc:2369] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:29:21.588817 26218 tablet.cc:2369] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:29:21.722543 25902 tablet.cc:2369] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:29:21.728431 26245 raft_consensus.cc:482] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:29:21.728786 26245 raft_consensus.cc:504] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8a5ea2ae6b459fa2a55be41f186664" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37291 } } peers { permanent_uuid: "f84cff3210e74de89c162728314cc453" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33163 } } peers { permanent_uuid: "32d078ed11c9454e9ee09278f8af64bd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42045 } }
I20260807 11:29:21.730647 26245 leader_election.cc:288] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe8a5ea2ae6b459fa2a55be41f186664 (127.24.103.67:37291), 32d078ed11c9454e9ee09278f8af64bd (127.24.103.65:42045)
I20260807 11:29:21.739101 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.731293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:45191 (local address 127.24.103.67:37291)
0807 11:29:21.731572 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:21.731581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:21.732540 (+ 959us) server_negotiation.cc:407] Connection header received
0807 11:29:21.732779 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:21.732786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:21.732868 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:21.732980 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:21.733634 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.734467 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.736975 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.737980 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.738146 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.738242 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:21.738665 (+ 423us) server_negotiation.cc:299] Negotiation successful
0807 11:29:21.738797 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 11:29:21.739117 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.731196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:55187 (local address 127.24.103.65:42045)
0807 11:29:21.731437 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:21.731446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:21.732646 (+ 1200us) server_negotiation.cc:407] Connection header received
0807 11:29:21.732874 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:21.732879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:21.732952 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:21.733053 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:21.733733 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.734465 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.736864 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.737992 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.738204 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.738285 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:21.738653 (+ 368us) server_negotiation.cc:299] Negotiation successful
0807 11:29:21.738796 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:29:21.739300 26249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.731251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42045 (local address 127.24.103.66:55187)
0807 11:29:21.732521 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:21.732541 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:21.732706 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:21.733067 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:21.733072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:21.733100 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:21.733568 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.733576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.734609 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.734615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.736673 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.736689 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.738279 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.738286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.738446 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.738503 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 11:29:21.738619 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 11:29:21.738755 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"mutex_wait_us":32,"thread_start_us":776,"threads_started":1}
I20260807 11:29:21.739342 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.730928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37291 (local address 127.24.103.66:45191)
0807 11:29:21.732412 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:21.732430 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:21.732592 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:21.732997 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:21.733002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:21.733028 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:21.733462 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.733472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.734617 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.734626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.736714 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:21.736728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.738243 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.738250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:21.738446 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.738501 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 11:29:21.738633 (+ 132us) client_negotiation.cc:240] Negotiation successful
0807 11:29:21.738777 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":45,"thread_start_us":822,"threads_started":1}
I20260807 11:29:21.740527 26162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e255c98e5572464dbc9e6dad99ef4215" candidate_uuid: "f84cff3210e74de89c162728314cc453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8a5ea2ae6b459fa2a55be41f186664" is_pre_election: true
I20260807 11:29:21.740576 25849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e255c98e5572464dbc9e6dad99ef4215" candidate_uuid: "f84cff3210e74de89c162728314cc453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d078ed11c9454e9ee09278f8af64bd" is_pre_election: true
I20260807 11:29:21.741084 26162 raft_consensus.cc:2454] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84cff3210e74de89c162728314cc453 in term 0.
I20260807 11:29:21.741218 25849 raft_consensus.cc:2454] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84cff3210e74de89c162728314cc453 in term 0.
I20260807 11:29:21.741798 25929 leader_election.cc:302] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f84cff3210e74de89c162728314cc453, fe8a5ea2ae6b459fa2a55be41f186664; no voters:
I20260807 11:29:21.742255 26245 raft_consensus.cc:2781] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:29:21.742413 26245 raft_consensus.cc:482] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:29:21.742549 26245 raft_consensus.cc:3037] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:21.745088 26245 raft_consensus.cc:504] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8a5ea2ae6b459fa2a55be41f186664" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37291 } } peers { permanent_uuid: "f84cff3210e74de89c162728314cc453" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33163 } } peers { permanent_uuid: "32d078ed11c9454e9ee09278f8af64bd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42045 } }
I20260807 11:29:21.745862 26245 leader_election.cc:288] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [CANDIDATE]: Term 1 election: Requested vote from peers fe8a5ea2ae6b459fa2a55be41f186664 (127.24.103.67:37291), 32d078ed11c9454e9ee09278f8af64bd (127.24.103.65:42045)
I20260807 11:29:21.746387 25849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e255c98e5572464dbc9e6dad99ef4215" candidate_uuid: "f84cff3210e74de89c162728314cc453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d078ed11c9454e9ee09278f8af64bd"
I20260807 11:29:21.746376 26162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e255c98e5572464dbc9e6dad99ef4215" candidate_uuid: "f84cff3210e74de89c162728314cc453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8a5ea2ae6b459fa2a55be41f186664"
I20260807 11:29:21.746622 26162 raft_consensus.cc:3037] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:21.746618 25849 raft_consensus.cc:3037] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:21.750339 26162 raft_consensus.cc:2454] T e255c98e5572464dbc9e6dad99ef4215 P fe8a5ea2ae6b459fa2a55be41f186664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84cff3210e74de89c162728314cc453 in term 1.
I20260807 11:29:21.750342 25849 raft_consensus.cc:2454] T e255c98e5572464dbc9e6dad99ef4215 P 32d078ed11c9454e9ee09278f8af64bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84cff3210e74de89c162728314cc453 in term 1.
I20260807 11:29:21.750811 25926 leader_election.cc:302] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d078ed11c9454e9ee09278f8af64bd, f84cff3210e74de89c162728314cc453; no voters:
I20260807 11:29:21.751175 26245 raft_consensus.cc:2781] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:21.751737 26245 raft_consensus.cc:686] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [term 1 LEADER]: Becoming Leader. State: Replica: f84cff3210e74de89c162728314cc453, State: Running, Role: LEADER
I20260807 11:29:21.752290 26245 consensus_queue.cc:237] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8a5ea2ae6b459fa2a55be41f186664" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37291 } } peers { permanent_uuid: "f84cff3210e74de89c162728314cc453" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33163 } } peers { permanent_uuid: "32d078ed11c9454e9ee09278f8af64bd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42045 } }
I20260807 11:29:21.757658 25681 catalog_manager.cc:5495] T e255c98e5572464dbc9e6dad99ef4215 P f84cff3210e74de89c162728314cc453 reported cstate change: term changed from 0 to 1, leader changed from <none> to f84cff3210e74de89c162728314cc453 (127.24.103.66). New cstate: current_term: 1 leader_uuid: "f84cff3210e74de89c162728314cc453" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8a5ea2ae6b459fa2a55be41f186664" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f84cff3210e74de89c162728314cc453" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32d078ed11c9454e9ee09278f8af64bd" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42045 } health_report { overall_health: UNKNOWN } } }
Aug 07 11:29:21 dist-test-slave-wl2h krb5kdc[25632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:29:21.828064 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.811210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.24.103.65:42045)
0807 11:29:21.811392 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:21.811397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:21.812076 (+ 679us) server_negotiation.cc:407] Connection header received
0807 11:29:21.812310 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:21.812315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:21.812384 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:21.812509 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:21.814109 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.814865 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.816164 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.816571 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.816748 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.821474 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:21.821492 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:21.821509 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:21.821548 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:21.824960 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.825740 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.825748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.825751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.825895 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.826322 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.826326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.826329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.826704 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:21.827060 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:21.827376 (+ 316us) server_negotiation.cc:299] Negotiation successful
0807 11:29:21.827636 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 11:29:21 dist-test-slave-wl2h krb5kdc[25632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 11:29:21 dist-test-slave-wl2h krb5kdc[25632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:29:21.884196 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.868334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.24.103.65:42045)
0807 11:29:21.868543 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:21.868551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:21.868930 (+ 379us) server_negotiation.cc:407] Connection header received
0807 11:29:21.869160 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:21.869164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:21.869235 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:21.869347 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:21.870803 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.871624 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.872932 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.873344 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.873483 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.878378 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:21.878383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:21.878387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:21.878427 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:21.881352 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.882041 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.882045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.882047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.882175 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.882653 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.882656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.882658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.882935 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:21.883287 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:21.883609 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 11:29:21.883867 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 11:29:21 dist-test-slave-wl2h krb5kdc[25632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 11:29:21 dist-test-slave-wl2h krb5kdc[25632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:29:21.939440 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.923907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.24.103.65:42045)
0807 11:29:21.924144 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:21.924158 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:21.924329 (+ 171us) server_negotiation.cc:407] Connection header received
0807 11:29:21.924489 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:21.924493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:21.924560 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:21.924652 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:21.926278 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.927003 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.928453 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.928927 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.929060 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.933836 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:21.933841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:21.933845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:21.933884 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:21.936638 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.937435 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.937440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.937442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.937604 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.938041 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.938045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.938048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.938373 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:21.938567 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:21.938835 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 11:29:21.939070 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260807 11:29:21.941399 25804 tablet_service.cc:2908] Rejecting scan request for tablet e255c98e5572464dbc9e6dad99ef4215: Uninitialized: clean time has not yet been initialized
I20260807 11:29:21.944950 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 25754
I20260807 11:29:21.958349 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:21.945976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.24.103.65:42045)
0807 11:29:21.946219 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:21.946225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:21.946499 (+ 274us) server_negotiation.cc:407] Connection header received
0807 11:29:21.946635 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:21.946640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:21.946730 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:21.946817 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:21.948239 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.948920 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.950151 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:21.950617 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:21.950732 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:21.953203 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:21.953208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:21.953212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:21.953245 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:21.956023 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.956782 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.956786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.956788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.956911 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:21.957225 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:21.957228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:21.957230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:21.957495 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:21.957716 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:21.957948 (+ 232us) server_negotiation.cc:299] Negotiation successful
0807 11:29:21.958122 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:29:21.959101 25895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:22.075138 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 25754
I20260807 11:29:22.095634 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 25908
Aug 07 11:29:22 dist-test-slave-wl2h krb5kdc[25632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:29:22.111605 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:22.096458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.24.103.66:33163)
0807 11:29:22.097027 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:22.097031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:22.097048 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:29:22.097105 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:22.097109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:22.097169 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:22.097309 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:22.098520 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:22.099350 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:22.100647 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:22.101085 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:22.101242 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:22.105338 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:22.105353 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:22.105369 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:22.105409 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:22.108442 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:22.109251 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:22.109257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:22.109259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:22.109418 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:22.109888 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:22.109893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:22.109895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:22.110301 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:22.110553 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:22.110854 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 11:29:22.111128 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":314,"threads_started":1}
I20260807 11:29:22.112373 26049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:22.208958 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 25908
I20260807 11:29:22.230409 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 26063
Aug 07 11:29:22 dist-test-slave-wl2h krb5kdc[25632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:29:22.247648 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:22.231295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.24.103.67:37291)
0807 11:29:22.231540 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:22.231546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:22.231783 (+ 237us) server_negotiation.cc:407] Connection header received
0807 11:29:22.232034 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:22.232038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:22.232149 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:22.232273 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:22.233648 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:22.234388 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:22.235880 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:22.236287 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:22.236427 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:22.241074 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:22.241093 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:22.241119 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:22.241160 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:22.244357 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:22.245240 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:22.245250 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:22.245252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:22.245420 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:22.245894 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:22.245898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:22.245900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:22.246312 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:22.246560 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:22.246955 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 11:29:22.247328 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:29:22.248423 26212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:22.338853 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 26063
I20260807 11:29:22.359929 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 25646
I20260807 11:29:22.361127 25717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:22.473920 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 25646
2026-08-07T11:29:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4765 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:29:22 dist-test-slave-wl2h krb5kdc[26272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:29:22 dist-test-slave-wl2h krb5kdc[26272](info): set up 2 sockets
Aug 07 11:29:22 dist-test-slave-wl2h krb5kdc[26272](info): commencing operation
krb5kdc: starting...
W20260807 11:29:23.671749 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:29:23 dist-test-slave-wl2h krb5kdc[26272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:29:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:29:23.902782 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:33723
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:24.238189 26290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:24.238574 26290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:24.238756 26290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:24.245970 26290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:24.246129 26290 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 11:29:24.246227 26290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:24.246340 26290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:24.246433 26290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:24.246520 26290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:29:24.256563 26290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:33723
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:24.257970 26290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:24.259683 26290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:24.272440 26298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:24.272536 26296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:24.272552 26301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:24.273639 26290 server_base.cc:1034] running on GCE node
I20260807 11:29:24.274566 26290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:24.276266 26290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:24.277561 26290 hybrid_clock.cc:648] HybridClock initialized: now 1786102164277462 us; error 55 us; skew 500 ppm
I20260807 11:29:24.277989 26290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:24 dist-test-slave-wl2h krb5kdc[26272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102164, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:24.281839 26290 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:29:24.283679 26290 webserver.cc:460] Webserver started at http://127.24.103.126:46263/ using document root <none> and password file <none>
I20260807 11:29:24.284477 26290 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:24.284583 26290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:24.284906 26290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:24.288427 26290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "c0fb1a7fbf5a4bd9b1c3c61fc363805e"
format_stamp: "Formatted at 2026-08-07 11:29:24 on dist-test-slave-wl2h"
server_key: "dd947a697ddbf52bc8de47b61bc1c4d7"
server_key_iv: "ad175ce3f5e26d3221e5d7694c2481c4"
server_key_version: "encryptionkey@0"
I20260807 11:29:24.289361 26290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "c0fb1a7fbf5a4bd9b1c3c61fc363805e"
format_stamp: "Formatted at 2026-08-07 11:29:24 on dist-test-slave-wl2h"
server_key: "dd947a697ddbf52bc8de47b61bc1c4d7"
server_key_iv: "ad175ce3f5e26d3221e5d7694c2481c4"
server_key_version: "encryptionkey@0"
I20260807 11:29:24.296370 26290 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260807 11:29:24.300835 26309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:24.302577 26290 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:29:24.302857 26290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "c0fb1a7fbf5a4bd9b1c3c61fc363805e"
format_stamp: "Formatted at 2026-08-07 11:29:24 on dist-test-slave-wl2h"
server_key: "dd947a697ddbf52bc8de47b61bc1c4d7"
server_key_iv: "ad175ce3f5e26d3221e5d7694c2481c4"
server_key_version: "encryptionkey@0"
I20260807 11:29:24.303211 26290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:24.324548 26290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:24.329454 26290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:24.329787 26290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:24.354532 26290 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:33723
I20260807 11:29:24.354550 26381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:33723 every 8 connection(s)
I20260807 11:29:24.356125 26290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:24.356976 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 26290
I20260807 11:29:24.357295 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:24.358038 24989 external_mini_cluster.cc:1428] Setting key f7be504357f1df01e2f46d9c31ebeefd
I20260807 11:29:24.360846 26382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:29:24 dist-test-slave-wl2h krb5kdc[26272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:24.374439 26382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e: Bootstrap starting.
I20260807 11:29:24.378964 26382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:24.380333 26382 log.cc:824] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:24.383774 26382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e: No bootstrap required, opened a new log
I20260807 11:29:24.384236 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:24.360265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.24.103.126:33723)
0807 11:29:24.361143 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:24.361160 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:24.361213 (+ 53us) server_negotiation.cc:407] Connection header received
0807 11:29:24.362981 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:24.363005 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:24.363513 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:24.363866 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:24.365738 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:24.367389 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:24.369115 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:24.369858 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:24.370004 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:24.375149 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:24.375161 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:24.375192 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:24.375245 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:24.379677 (+ 4432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:24.380590 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:24.380606 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:24.380613 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:24.380802 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:24.381241 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:24.381249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:24.381257 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:24.381861 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:24.382170 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:24.382688 (+ 518us) server_negotiation.cc:299] Negotiation successful
0807 11:29:24.383161 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":433,"threads_started":1}
I20260807 11:29:24.392284 26382 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fb1a7fbf5a4bd9b1c3c61fc363805e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33723 } }
I20260807 11:29:24.392699 26382 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:24.392822 26382 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fb1a7fbf5a4bd9b1c3c61fc363805e, State: Initialized, Role: FOLLOWER
I20260807 11:29:24.393453 26382 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fb1a7fbf5a4bd9b1c3c61fc363805e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33723 } }
I20260807 11:29:24.393746 26382 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:24.393913 26382 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:24.394109 26382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:24.396433 26382 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fb1a7fbf5a4bd9b1c3c61fc363805e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33723 } }
I20260807 11:29:24.396986 26382 leader_election.cc:302] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0fb1a7fbf5a4bd9b1c3c61fc363805e; no voters:
I20260807 11:29:24.397619 26382 leader_election.cc:288] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:24.397835 26387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:24.398849 26387 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [term 1 LEADER]: Becoming Leader. State: Replica: c0fb1a7fbf5a4bd9b1c3c61fc363805e, State: Running, Role: LEADER
I20260807 11:29:24.399544 26387 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fb1a7fbf5a4bd9b1c3c61fc363805e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33723 } }
I20260807 11:29:24.400329 26382 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:24.402698 26390 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0fb1a7fbf5a4bd9b1c3c61fc363805e. Latest consensus state: current_term: 1 leader_uuid: "c0fb1a7fbf5a4bd9b1c3c61fc363805e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fb1a7fbf5a4bd9b1c3c61fc363805e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33723 } } }
I20260807 11:29:24.402722 26389 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0fb1a7fbf5a4bd9b1c3c61fc363805e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fb1a7fbf5a4bd9b1c3c61fc363805e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33723 } } }
I20260807 11:29:24.403287 26389 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:24.403294 26390 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:24.405750 26395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:24.412457 26395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:24.421744 26395 catalog_manager.cc:1347] Generated new cluster ID: bbb9eb219e304f538dc2c50f7c254435
I20260807 11:29:24.421921 26395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:24.431732 26395 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:24.433763 26395 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:24.451609 26395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e: Generated new TSK 0
I20260807 11:29:24.452625 26395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:29:24.522657 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33723
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:24.881027 26412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:24.881325 26412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:24.881424 26412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:24.889387 26412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:24.889521 26412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:24.889611 26412 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:29:24.889698 26412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:29:24.900321 26412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:33723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:24.901746 26412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:24.903177 26412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:24.916896 26421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:24.917074 26419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:24.917297 26418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:24.918322 26412 server_base.cc:1034] running on GCE node
I20260807 11:29:24.919160 26412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:24.920416 26412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:24.921721 26412 hybrid_clock.cc:648] HybridClock initialized: now 1786102164921637 us; error 39 us; skew 500 ppm
I20260807 11:29:24.922168 26412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:24 dist-test-slave-wl2h krb5kdc[26272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102164, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:24.925604 26412 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:29:24.927500 26412 webserver.cc:460] Webserver started at http://127.24.103.65:45909/ using document root <none> and password file <none>
I20260807 11:29:24.928294 26412 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:24.928436 26412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:24.928810 26412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:24.932091 26412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "76096aec61c24e2fb51ae2d18af026c2"
format_stamp: "Formatted at 2026-08-07 11:29:24 on dist-test-slave-wl2h"
server_key: "34fea5cc8115ac29b095e516ed0cb722"
server_key_iv: "19c56d2ae75f6ddbb601b81ebbb99368"
server_key_version: "encryptionkey@0"
I20260807 11:29:24.933020 26412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "76096aec61c24e2fb51ae2d18af026c2"
format_stamp: "Formatted at 2026-08-07 11:29:24 on dist-test-slave-wl2h"
server_key: "34fea5cc8115ac29b095e516ed0cb722"
server_key_iv: "19c56d2ae75f6ddbb601b81ebbb99368"
server_key_version: "encryptionkey@0"
I20260807 11:29:24.938480 26412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 11:29:24.941995 26432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:24.943351 26412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:24.943531 26412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "76096aec61c24e2fb51ae2d18af026c2"
format_stamp: "Formatted at 2026-08-07 11:29:24 on dist-test-slave-wl2h"
server_key: "34fea5cc8115ac29b095e516ed0cb722"
server_key_iv: "19c56d2ae75f6ddbb601b81ebbb99368"
server_key_version: "encryptionkey@0"
I20260807 11:29:24.943782 26412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:24.960304 26412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:24.964797 26412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:24.965098 26412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:24.968303 26412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:24.968499 26412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260807 11:29:24.968691 26412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:24.968801 26412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:29:24 dist-test-slave-wl2h krb5kdc[26272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102164, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:24.994498 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:24.968683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:54429 (local address 127.24.103.126:33723)
0807 11:29:24.969223 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:24.969230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:24.970415 (+ 1185us) server_negotiation.cc:407] Connection header received
0807 11:29:24.972587 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:24.972596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:24.972721 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:24.972852 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:24.975208 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:24.976578 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:24.979513 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:24.980293 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:24.980470 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:24.986555 (+ 6085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:24.986565 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:24.986570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:24.986625 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:24.990155 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:24.991379 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:24.991387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:24.991390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:24.991539 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:24.992254 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:24.992260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:24.992266 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:24.992644 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:24.992810 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:24.993871 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0807 11:29:24.994111 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
I20260807 11:29:24.995622 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:24.969310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33723 (local address 127.24.103.65:54429)
0807 11:29:24.970220 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:24.970271 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:24.972278 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:24.973112 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:24.973121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:24.973982 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:24.974938 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:24.974974 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:24.976819 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:24.976831 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:24.979365 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:24.979379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:24.980624 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:24.980633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:24.980883 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:24.981968 (+ 1085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:24.982013 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:24.986156 (+ 4143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:24.990337 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:24.990354 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:24.990376 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:24.991171 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:24.991666 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:24.991672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:24.991678 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:24.992035 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:24.992896 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:24.992905 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:24.993437 (+ 532us) client_negotiation.cc:769] Sending connection context
0807 11:29:24.993756 (+ 319us) client_negotiation.cc:240] Negotiation successful
0807 11:29:24.994340 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":392,"threads_started":1}
I20260807 11:29:25.019459 26571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:35305 every 8 connection(s)
I20260807 11:29:25.019459 26412 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:35305
I20260807 11:29:25.020939 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:25.002696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:58121 (local address 127.24.103.126:33723)
0807 11:29:25.002963 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:25.002973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:25.003673 (+ 700us) server_negotiation.cc:407] Connection header received
0807 11:29:25.003929 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:25.003938 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:25.004046 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:25.004185 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:25.006484 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.007675 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.009731 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.010248 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.010404 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:25.012798 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:25.012807 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:25.012814 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:25.012868 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:25.017273 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:25.018245 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:25.018256 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:25.018263 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:25.018497 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:25.019059 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:25.019066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:25.019070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:25.019541 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:25.019754 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:25.020284 (+ 530us) server_negotiation.cc:299] Negotiation successful
0807 11:29:25.020541 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 11:29:25.020979 26412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:29:25.021050 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:25.002641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33723 (local address 127.24.103.65:58121)
0807 11:29:25.003514 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:25.003555 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:25.003763 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:25.004272 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:25.004280 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:25.005162 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:25.006176 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:25.006207 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.007905 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.007916 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:25.009485 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:25.009509 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.010471 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.010480 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:25.010728 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:25.011661 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:25.011685 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:25.012559 (+ 874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:25.017432 (+ 4873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:25.017438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:25.017442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:25.018033 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:25.018641 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:25.018646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:25.018650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:25.018869 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:25.019684 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:25.019690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:25.019917 (+ 227us) client_negotiation.cc:769] Sending connection context
0807 11:29:25.020095 (+ 178us) client_negotiation.cc:240] Negotiation successful
0807 11:29:25.020505 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":473,"threads_started":1}
I20260807 11:29:25.023706 26572 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33723
I20260807 11:29:25.024035 26572 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:25.024873 26572 heartbeater.cc:511] Master 127.24.103.126:33723 requested a full tablet report, sending...
I20260807 11:29:25.027237 26331 ts_manager.cc:194] Registered new tserver with Master: 76096aec61c24e2fb51ae2d18af026c2 (127.24.103.65:35305)
I20260807 11:29:25.028824 26331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:54429
I20260807 11:29:25.029016 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 26412
I20260807 11:29:25.029273 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:29:25.029963 24989 external_mini_cluster.cc:1428] Setting key 1ed48fe6ab3f86039abfcf3cc7269d08
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:29:25.078670 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33723
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:25.421670 26576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:25.421962 26576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:25.422137 26576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:25.429530 26576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:25.429643 26576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:25.429728 26576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:29:25.429812 26576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:29:25.439606 26576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:33723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:25.440846 26576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:25.442171 26576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:25.455677 26587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:25.455657 26588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:25.455677 26591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:25.456419 26576 server_base.cc:1034] running on GCE node
I20260807 11:29:25.457019 26576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:25.457938 26576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:25.459178 26576 hybrid_clock.cc:648] HybridClock initialized: now 1786102165459122 us; error 32 us; skew 500 ppm
I20260807 11:29:25.459484 26576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:25 dist-test-slave-wl2h krb5kdc[26272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102165, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:25.462589 26576 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:29:25.464066 26576 webserver.cc:460] Webserver started at http://127.24.103.66:32977/ using document root <none> and password file <none>
I20260807 11:29:25.464710 26576 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:25.464797 26576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:25.465063 26576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:25.469398 26576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "9e7336962baf4eefa9b4dceb43484319"
format_stamp: "Formatted at 2026-08-07 11:29:25 on dist-test-slave-wl2h"
server_key: "db18b5ba3aab336b3a34bab2bb950f9d"
server_key_iv: "76b608e4ae07fca3fce8ad5151e4b20a"
server_key_version: "encryptionkey@0"
I20260807 11:29:25.470253 26576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "9e7336962baf4eefa9b4dceb43484319"
format_stamp: "Formatted at 2026-08-07 11:29:25 on dist-test-slave-wl2h"
server_key: "db18b5ba3aab336b3a34bab2bb950f9d"
server_key_iv: "76b608e4ae07fca3fce8ad5151e4b20a"
server_key_version: "encryptionkey@0"
I20260807 11:29:25.475817 26576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 11:29:25.479400 26598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:25.480808 26576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:29:25.480995 26576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "9e7336962baf4eefa9b4dceb43484319"
format_stamp: "Formatted at 2026-08-07 11:29:25 on dist-test-slave-wl2h"
server_key: "db18b5ba3aab336b3a34bab2bb950f9d"
server_key_iv: "76b608e4ae07fca3fce8ad5151e4b20a"
server_key_version: "encryptionkey@0"
I20260807 11:29:25.481248 26576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:25.517324 26576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:25.521512 26576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:25.521795 26576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:25.524600 26576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:25.524724 26576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:25.524890 26576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:25.524991 26576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:29:25 dist-test-slave-wl2h krb5kdc[26272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102165, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:25.550726 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:25.525233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:56925 (local address 127.24.103.126:33723)
0807 11:29:25.525996 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:25.526005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:25.527114 (+ 1109us) server_negotiation.cc:407] Connection header received
0807 11:29:25.530220 (+ 3106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:25.530227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:25.530331 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:25.530445 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:25.533326 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.534435 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.537102 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.537941 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.538103 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:25.543347 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:25.543353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:25.543356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:25.543393 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:25.547103 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:25.548096 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:25.548102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:25.548105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:25.548251 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:25.548733 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:25.548739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:25.548742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:25.549027 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:25.549311 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:25.550094 (+ 783us) server_negotiation.cc:299] Negotiation successful
0807 11:29:25.550332 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":506,"threads_started":1}
I20260807 11:29:25.552266 26612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:25.525933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33723 (local address 127.24.103.66:56925)
0807 11:29:25.526896 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:25.526962 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:25.529820 (+ 2858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:25.531161 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:25.531177 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:25.532125 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:25.533094 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:25.533124 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.534659 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.534663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:25.536908 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:25.536936 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.538243 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.538252 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:25.538535 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:25.539335 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:25.539401 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:25.543062 (+ 3661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:25.547268 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:25.547278 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:25.547296 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:25.547954 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:25.548372 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:25.548377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:25.548381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:25.548602 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:25.549365 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:25.549372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:25.549718 (+ 346us) client_negotiation.cc:769] Sending connection context
0807 11:29:25.549995 (+ 277us) client_negotiation.cc:240] Negotiation successful
0807 11:29:25.550544 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":426,"threads_started":1}
I20260807 11:29:25.575961 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:25.558645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:56175 (local address 127.24.103.126:33723)
0807 11:29:25.558834 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:25.558840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:25.559185 (+ 345us) server_negotiation.cc:407] Connection header received
0807 11:29:25.559393 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:25.559396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:25.559460 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:25.559540 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:25.561019 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.561759 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.564312 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.564908 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.565052 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:25.568030 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:25.568039 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:25.568047 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:25.568096 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:25.572289 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:25.573054 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:25.573060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:25.573064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:25.573249 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:25.573817 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:25.573826 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:25.573832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:25.574371 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:25.574570 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:25.575228 (+ 658us) server_negotiation.cc:299] Negotiation successful
0807 11:29:25.575525 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 11:29:25.576191 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:25.558306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33723 (local address 127.24.103.66:56175)
0807 11:29:25.559066 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:25.559091 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:25.559270 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:25.559606 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:25.559613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:25.560359 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:25.560842 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:25.560858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.561909 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.561918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:25.564141 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:25.564159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:25.565171 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:25.565182 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:25.565384 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:25.566134 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:25.566161 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:25.567748 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:25.572446 (+ 4698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:25.572450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:25.572453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:25.572901 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:25.573432 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:25.573444 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:25.573449 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:25.573657 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:25.574520 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:25.574527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:25.574847 (+ 320us) client_negotiation.cc:769] Sending connection context
0807 11:29:25.575071 (+ 224us) client_negotiation.cc:240] Negotiation successful
0807 11:29:25.575547 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":412,"threads_started":1}
I20260807 11:29:25.582343 26576 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:39455
I20260807 11:29:25.582439 26739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:39455 every 8 connection(s)
I20260807 11:29:25.583848 26576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:29:25.586315 26740 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33723
I20260807 11:29:25.586678 26740 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:25.587447 26740 heartbeater.cc:511] Master 127.24.103.126:33723 requested a full tablet report, sending...
I20260807 11:29:25.588871 26331 ts_manager.cc:194] Registered new tserver with Master: 9e7336962baf4eefa9b4dceb43484319 (127.24.103.66:39455)
I20260807 11:29:25.589721 26331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:56925
I20260807 11:29:25.592370 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 26576
I20260807 11:29:25.592589 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:29:25.593022 24989 external_mini_cluster.cc:1428] Setting key f1329f9010811941101e909891bf25b7
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:29:25.643114 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33723
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:25.992615 26744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:25.992916 26744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:25.993089 26744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:26.000883 26744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:26.001017 26744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:26.001164 26744 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:29:26.001279 26744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:29:26.012481 26744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:33723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:26.013996 26744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:26.015517 26744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:26.030833 26749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:26.031080 26751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:26.031134 26754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:26.032085 26744 server_base.cc:1034] running on GCE node
I20260807 11:29:26.032285 26572 heartbeater.cc:503] Master 127.24.103.126:33723 was elected leader, sending a full tablet report...
I20260807 11:29:26.032893 26744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:26.033994 26744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:26.035259 26744 hybrid_clock.cc:648] HybridClock initialized: now 1786102166035164 us; error 59 us; skew 500 ppm
I20260807 11:29:26.035575 26744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:26 dist-test-slave-wl2h krb5kdc[26272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102166, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:26.039067 26744 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:29:26.040779 26744 webserver.cc:460] Webserver started at http://127.24.103.67:46265/ using document root <none> and password file <none>
I20260807 11:29:26.041565 26744 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:26.041669 26744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:26.042068 26744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:26.045594 26744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "20484d6c9fbc47caba26c1f2b994eeaf"
format_stamp: "Formatted at 2026-08-07 11:29:26 on dist-test-slave-wl2h"
server_key: "a41584a1a73ac6cfdcceffbdbbecd545"
server_key_iv: "a2760470720d2c2044228552a0132b1c"
server_key_version: "encryptionkey@0"
I20260807 11:29:26.046577 26744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "20484d6c9fbc47caba26c1f2b994eeaf"
format_stamp: "Formatted at 2026-08-07 11:29:26 on dist-test-slave-wl2h"
server_key: "a41584a1a73ac6cfdcceffbdbbecd545"
server_key_iv: "a2760470720d2c2044228552a0132b1c"
server_key_version: "encryptionkey@0"
I20260807 11:29:26.053258 26744 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:29:26.056969 26764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:26.058538 26744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:29:26.058769 26744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "20484d6c9fbc47caba26c1f2b994eeaf"
format_stamp: "Formatted at 2026-08-07 11:29:26 on dist-test-slave-wl2h"
server_key: "a41584a1a73ac6cfdcceffbdbbecd545"
server_key_iv: "a2760470720d2c2044228552a0132b1c"
server_key_version: "encryptionkey@0"
I20260807 11:29:26.059096 26744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:26.082880 26744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:26.088656 26744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:26.088996 26744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:26.091988 26744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:26.092162 26744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:26.092325 26744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:26.092402 26744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:29:26 dist-test-slave-wl2h krb5kdc[26272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102166, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:26.121336 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.093003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:50881 (local address 127.24.103.126:33723)
0807 11:29:26.093701 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.093706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.095156 (+ 1450us) server_negotiation.cc:407] Connection header received
0807 11:29:26.098136 (+ 2980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.098144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.098254 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.098393 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:26.100720 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.101937 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.105149 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.105654 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.105823 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.111717 (+ 5894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:26.111724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:26.111727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:26.111766 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:26.115948 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:26.117260 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:26.117267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:26.117272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:26.117494 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:26.118269 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:26.118279 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:26.118285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:26.118853 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:26.119088 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.120231 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.120667 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":338,"threads_started":1}
I20260807 11:29:26.121904 26778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.093752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33723 (local address 127.24.103.67:50881)
0807 11:29:26.094879 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.094936 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.097731 (+ 2795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.098712 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.098729 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.099757 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:26.100480 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.100500 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.102154 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.102162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.104910 (+ 2748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.104931 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.106006 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.106016 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.106446 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.107615 (+ 1169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:26.107647 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:26.111421 (+ 3774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:26.116214 (+ 4793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:26.116225 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:26.116240 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:26.117056 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:26.117736 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:26.117742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:26.117746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:26.118109 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:26.119090 (+ 981us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:26.119102 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:26.119665 (+ 563us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.120075 (+ 410us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.120626 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":435,"threads_started":1}
I20260807 11:29:26.144335 26744 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:46609
I20260807 11:29:26.144381 26908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:46609 every 8 connection(s)
I20260807 11:29:26.146059 26744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:29:26.146780 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.129459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:33007 (local address 127.24.103.126:33723)
0807 11:29:26.129703 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.129712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.131572 (+ 1860us) server_negotiation.cc:407] Connection header received
0807 11:29:26.131929 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.131939 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.132071 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.132167 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:26.134198 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.134944 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.137084 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.137664 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.137823 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.140468 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:26.140472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:26.140476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:26.140508 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:26.143800 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:26.144611 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:26.144616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:26.144620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:26.144762 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:26.145208 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:26.145211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:26.145213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:26.145492 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:26.145618 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.146302 (+ 684us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.146486 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 11:29:26.147249 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.129468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33723 (local address 127.24.103.67:33007)
0807 11:29:26.131399 (+ 1931us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.131432 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.131663 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.132493 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.132501 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.133465 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:26.134018 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.134034 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.135163 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.135168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.136925 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.136941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.137873 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.137879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.138119 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.138983 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:26.139036 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:26.140214 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:26.143939 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:26.143944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:26.143948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:26.144438 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:26.144893 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:26.144898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:26.144902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:26.145077 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:26.145653 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:26.145659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:26.145883 (+ 224us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.146082 (+ 199us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.146473 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1479,"thread_start_us":829,"threads_started":1}
I20260807 11:29:26.148514 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 26744
I20260807 11:29:26.148847 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:29:26.149643 24989 external_mini_cluster.cc:1428] Setting key 8e3fae8b8d10ece5f6e4d59791c6ff6f
I20260807 11:29:26.150374 26909 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33723
I20260807 11:29:26.150635 26909 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:26.151670 26909 heartbeater.cc:511] Master 127.24.103.126:33723 requested a full tablet report, sending...
I20260807 11:29:26.153450 26330 ts_manager.cc:194] Registered new tserver with Master: 20484d6c9fbc47caba26c1f2b994eeaf (127.24.103.67:46609)
I20260807 11:29:26.154904 26330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:50881
I20260807 11:29:26.166501 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:29:26 dist-test-slave-wl2h krb5kdc[26272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102166, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 11:29:26 dist-test-slave-wl2h krb5kdc[26272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102166, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:26.222992 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.207860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.24.103.126:33723)
0807 11:29:26.208093 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.208098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.208502 (+ 404us) server_negotiation.cc:407] Connection header received
0807 11:29:26.208748 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.208753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.208826 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.208927 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:26.210443 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.211282 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.212609 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.213078 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.213226 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.217556 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:26.217561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:26.217565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:26.217609 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:26.220463 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:26.221171 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:26.221175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:26.221177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:26.221301 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:26.221655 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:26.221663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:26.221666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:26.221961 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:26.222199 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.222563 (+ 364us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.222729 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 11:29:26.228165 26330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:29:26.235428 26330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:29:26.261009 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.248770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46609 (local address 127.0.0.1:50898)
0807 11:29:26.249798 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.249847 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.250084 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.252007 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.252036 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.252075 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:26.252539 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.252554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.255041 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.255046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.257830 (+ 2784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.257846 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.259876 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.259882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.260127 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.260203 (+ 76us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.260320 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.260561 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":114,"threads_started":1}
I20260807 11:29:26.261121 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.249425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.24.103.65:35305)
0807 11:29:26.250691 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.250704 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.251666 (+ 962us) server_negotiation.cc:407] Connection header received
0807 11:29:26.251972 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.251982 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.252230 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.252405 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:26.253371 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.254952 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.258024 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.259643 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.259765 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.260004 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.260602 (+ 598us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.260807 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":937,"threads_started":1}
I20260807 11:29:26.261958 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.248965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35305 (local address 127.0.0.1:33274)
0807 11:29:26.251546 (+ 2581us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.251575 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.251769 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.252669 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.252677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.252704 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:26.253153 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.253169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.255203 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.255209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.257835 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.257846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.260203 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.260209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.260390 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.260418 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.261390 (+ 972us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.261545 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2455,"mutex_wait_us":89,"thread_start_us":405,"threads_started":1}
I20260807 11:29:26.262478 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.249023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50898 (local address 127.24.103.67:46609)
0807 11:29:26.251346 (+ 2323us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.251355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.251393 (+ 38us) server_negotiation.cc:407] Connection header received
0807 11:29:26.251499 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.251509 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.251771 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.252761 (+ 990us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:26.253116 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.254844 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.258043 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.259642 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.261570 (+ 1928us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.261823 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.261906 (+ 83us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.262106 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2145,"thread_start_us":1906,"threads_started":1}
I20260807 11:29:26.266312 26492 tablet_service.cc:1467] Processing CreateTablet for tablet 52c4b6a97c564feaa5dbd73f5ad428f2 (DEFAULT_TABLE table=test-table [id=ee6ae4bf8cf546ad9b2564367efb9197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:26.266690 26830 tablet_service.cc:1467] Processing CreateTablet for tablet 52c4b6a97c564feaa5dbd73f5ad428f2 (DEFAULT_TABLE table=test-table [id=ee6ae4bf8cf546ad9b2564367efb9197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:26.267889 26492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c4b6a97c564feaa5dbd73f5ad428f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:26.268071 26830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c4b6a97c564feaa5dbd73f5ad428f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:26.270341 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.250944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:39455 (local address 127.0.0.1:50392)
0807 11:29:26.261282 (+ 10338us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.261304 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.261409 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.263812 (+ 2403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.263817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.263845 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:26.264015 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.264026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.265892 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.265898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.267732 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.267748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.269538 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.269545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.269740 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.269773 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.269877 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.270045 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10213,"thread_start_us":1207,"threads_started":1}
I20260807 11:29:26.270416 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.251958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.24.103.66:39455)
0807 11:29:26.263151 (+ 11193us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.263160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.263194 (+ 34us) server_negotiation.cc:407] Connection header received
0807 11:29:26.263300 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.263308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.263585 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.263860 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:26.264146 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.265672 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.267927 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.269335 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.269497 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.269714 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.269945 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.270111 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10524,"thread_start_us":10765,"threads_started":1}
I20260807 11:29:26.274369 26658 tablet_service.cc:1467] Processing CreateTablet for tablet 52c4b6a97c564feaa5dbd73f5ad428f2 (DEFAULT_TABLE table=test-table [id=ee6ae4bf8cf546ad9b2564367efb9197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:26.275980 26658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c4b6a97c564feaa5dbd73f5ad428f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:26.281800 26930 tablet_bootstrap.cc:492] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: Bootstrap starting.
I20260807 11:29:26.284066 26931 tablet_bootstrap.cc:492] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2: Bootstrap starting.
I20260807 11:29:26.286397 26930 tablet_bootstrap.cc:654] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:26.287953 26930 log.cc:824] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:26.288309 26931 tablet_bootstrap.cc:654] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:26.289830 26931 log.cc:824] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:26.291081 26934 tablet_bootstrap.cc:492] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: Bootstrap starting.
I20260807 11:29:26.294307 26934 tablet_bootstrap.cc:654] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:26.296113 26934 log.cc:824] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:26.297857 26930 tablet_bootstrap.cc:492] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: No bootstrap required, opened a new log
I20260807 11:29:26.297883 26931 tablet_bootstrap.cc:492] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2: No bootstrap required, opened a new log
I20260807 11:29:26.298198 26930 ts_tablet_manager.cc:1397] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260807 11:29:26.298240 26931 ts_tablet_manager.cc:1397] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260807 11:29:26.299609 26934 tablet_bootstrap.cc:492] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: No bootstrap required, opened a new log
I20260807 11:29:26.299877 26934 ts_tablet_manager.cc:1397] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260807 11:29:26.307700 26931 raft_consensus.cc:348] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } }
I20260807 11:29:26.308161 26931 raft_consensus.cc:374] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:26.308270 26931 raft_consensus.cc:729] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76096aec61c24e2fb51ae2d18af026c2, State: Initialized, Role: FOLLOWER
I20260807 11:29:26.308985 26931 consensus_queue.cc:260] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } }
I20260807 11:29:26.309365 26930 raft_consensus.cc:348] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } }
I20260807 11:29:26.309813 26930 raft_consensus.cc:374] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:26.309919 26930 raft_consensus.cc:729] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20484d6c9fbc47caba26c1f2b994eeaf, State: Initialized, Role: FOLLOWER
I20260807 11:29:26.310501 26930 consensus_queue.cc:260] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } }
I20260807 11:29:26.310621 26934 raft_consensus.cc:348] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } }
I20260807 11:29:26.311066 26934 raft_consensus.cc:374] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:26.311197 26934 raft_consensus.cc:729] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7336962baf4eefa9b4dceb43484319, State: Initialized, Role: FOLLOWER
I20260807 11:29:26.311932 26934 consensus_queue.cc:260] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } }
I20260807 11:29:26.312573 26931 ts_tablet_manager.cc:1428] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260807 11:29:26.313521 26909 heartbeater.cc:503] Master 127.24.103.126:33723 was elected leader, sending a full tablet report...
I20260807 11:29:26.314306 26930 ts_tablet_manager.cc:1428] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260807 11:29:26.317231 26740 heartbeater.cc:503] Master 127.24.103.126:33723 was elected leader, sending a full tablet report...
I20260807 11:29:26.317875 26934 ts_tablet_manager.cc:1428] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: Time spent starting tablet: real 0.018s user 0.015s sys 0.003s
W20260807 11:29:26.336045 26741 tablet.cc:2369] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:29:26.404811 26910 tablet.cc:2369] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:29:26.406200 26938 raft_consensus.cc:482] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:29:26.406608 26938 raft_consensus.cc:504] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } }
I20260807 11:29:26.408769 26938 leader_election.cc:288] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76096aec61c24e2fb51ae2d18af026c2 (127.24.103.65:35305), 9e7336962baf4eefa9b4dceb43484319 (127.24.103.66:39455)
I20260807 11:29:26.416981 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.409231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:51547 (local address 127.24.103.65:35305)
0807 11:29:26.409445 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.409450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.409471 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:29:26.409813 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.409818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.409884 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.410023 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:26.410912 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.411971 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.414689 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.415965 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.416148 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.416230 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.416641 (+ 411us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.416757 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:29:26.417400 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.409908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:50921 (local address 127.24.103.66:39455)
0807 11:29:26.410145 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.410150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.410536 (+ 386us) server_negotiation.cc:407] Connection header received
0807 11:29:26.410878 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.410883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.410970 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.411162 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:26.411628 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.412519 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.415118 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.416364 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.416489 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.416570 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.417016 (+ 446us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.417176 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 11:29:26.417438 26940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.409709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:39455 (local address 127.24.103.67:50921)
0807 11:29:26.410391 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.410416 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.410604 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.411145 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.411148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.411172 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:26.411451 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.411458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.412690 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.412696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.414889 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.414901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.416643 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.416651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.416819 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.416854 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.416967 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.417095 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":359,"threads_started":1}
I20260807 11:29:26.417474 26778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.409174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35305 (local address 127.24.103.67:51547)
0807 11:29:26.409394 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.409414 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.409574 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.410318 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.410324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.410356 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:26.410643 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.410661 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.412184 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.412188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.414469 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.414488 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.416221 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.416226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.416390 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.416455 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.416594 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.416754 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":1}
I20260807 11:29:26.418934 26519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c4b6a97c564feaa5dbd73f5ad428f2" candidate_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76096aec61c24e2fb51ae2d18af026c2" is_pre_election: true
I20260807 11:29:26.418934 26685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c4b6a97c564feaa5dbd73f5ad428f2" candidate_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e7336962baf4eefa9b4dceb43484319" is_pre_election: true
I20260807 11:29:26.419613 26519 raft_consensus.cc:2454] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20484d6c9fbc47caba26c1f2b994eeaf in term 0.
I20260807 11:29:26.419613 26685 raft_consensus.cc:2454] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20484d6c9fbc47caba26c1f2b994eeaf in term 0.
I20260807 11:29:26.420409 26766 leader_election.cc:302] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20484d6c9fbc47caba26c1f2b994eeaf, 76096aec61c24e2fb51ae2d18af026c2; no voters:
I20260807 11:29:26.420835 26938 raft_consensus.cc:2781] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:29:26.421015 26938 raft_consensus.cc:482] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:29:26.421149 26938 raft_consensus.cc:3037] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:26.423538 26938 raft_consensus.cc:504] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } }
I20260807 11:29:26.424234 26938 leader_election.cc:288] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [CANDIDATE]: Term 1 election: Requested vote from peers 76096aec61c24e2fb51ae2d18af026c2 (127.24.103.65:35305), 9e7336962baf4eefa9b4dceb43484319 (127.24.103.66:39455)
I20260807 11:29:26.424854 26519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c4b6a97c564feaa5dbd73f5ad428f2" candidate_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76096aec61c24e2fb51ae2d18af026c2"
I20260807 11:29:26.424937 26685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c4b6a97c564feaa5dbd73f5ad428f2" candidate_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e7336962baf4eefa9b4dceb43484319"
I20260807 11:29:26.425177 26519 raft_consensus.cc:3037] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:26.425156 26685 raft_consensus.cc:3037] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:26.427569 26685 raft_consensus.cc:2454] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20484d6c9fbc47caba26c1f2b994eeaf in term 1.
I20260807 11:29:26.427796 26519 raft_consensus.cc:2454] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20484d6c9fbc47caba26c1f2b994eeaf in term 1.
I20260807 11:29:26.428012 26766 leader_election.cc:302] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20484d6c9fbc47caba26c1f2b994eeaf, 9e7336962baf4eefa9b4dceb43484319; no voters:
I20260807 11:29:26.428344 26938 raft_consensus.cc:2781] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:26.428840 26938 raft_consensus.cc:686] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 LEADER]: Becoming Leader. State: Replica: 20484d6c9fbc47caba26c1f2b994eeaf, State: Running, Role: LEADER
I20260807 11:29:26.429342 26938 consensus_queue.cc:237] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } }
I20260807 11:29:26.434341 26330 catalog_manager.cc:5495] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 20484d6c9fbc47caba26c1f2b994eeaf (127.24.103.67). New cstate: current_term: 1 leader_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20260807 11:29:26.464872 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.458343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.24.103.67:46609)
0807 11:29:26.458610 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.458619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.458775 (+ 156us) server_negotiation.cc:407] Connection header received
0807 11:29:26.459192 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.459197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.459297 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.459460 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:26.459855 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.460644 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.462145 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.462657 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.462812 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.463495 (+ 683us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:26.464067 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:26.464310 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.464452 (+ 142us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.464548 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:29:26.474898 26519 raft_consensus.cc:1261] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2 [term 1 FOLLOWER]: Refusing update from remote peer 20484d6c9fbc47caba26c1f2b994eeaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:26.474888 26685 raft_consensus.cc:1261] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319 [term 1 FOLLOWER]: Refusing update from remote peer 20484d6c9fbc47caba26c1f2b994eeaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:26.475929 26938 consensus_queue.cc:1035] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:26.476279 26941 consensus_queue.cc:1035] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:26.489256 26946 mvcc.cc:205] Tried to move back new op lower bound from 7315874473869570048 to 7315874473698791424. Current Snapshot: MvccSnapshot[applied={T|T < 7315874473869570048}]
I20260807 11:29:26.510917 26331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:26.511322 26331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:26.515547 26331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e: Sending DeleteTablet for 3 replicas of tablet 52c4b6a97c564feaa5dbd73f5ad428f2
I20260807 11:29:26.517412 26830 tablet_service.cc:1514] Processing DeleteTablet for tablet 52c4b6a97c564feaa5dbd73f5ad428f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:26 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:50898
I20260807 11:29:26.517460 26492 tablet_service.cc:1514] Processing DeleteTablet for tablet 52c4b6a97c564feaa5dbd73f5ad428f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:26 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:33274
I20260807 11:29:26.518558 26658 tablet_service.cc:1514] Processing DeleteTablet for tablet 52c4b6a97c564feaa5dbd73f5ad428f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:26 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:50392
I20260807 11:29:26.518631 26331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:29:26.519210 26331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:29:26.527417 26959 tablet_replica.cc:330] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: stopping tablet replica
I20260807 11:29:26.527997 26959 raft_consensus.cc:2229] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:26.528546 26959 raft_consensus.cc:2258] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:26.530718 26658 tablet_service.cc:1467] Processing CreateTablet for tablet b3bea311fb784f60bb9970ace0504cca (DEFAULT_TABLE table=test-table [id=e480bf7d247242b8974c842a4e7c6bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:26.530884 26491 tablet_service.cc:1467] Processing CreateTablet for tablet b3bea311fb784f60bb9970ace0504cca (DEFAULT_TABLE table=test-table [id=e480bf7d247242b8974c842a4e7c6bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:26.531553 26658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bea311fb784f60bb9970ace0504cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:26.531569 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bea311fb784f60bb9970ace0504cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:26.532692 26829 tablet_service.cc:1467] Processing CreateTablet for tablet b3bea311fb784f60bb9970ace0504cca (DEFAULT_TABLE table=test-table [id=e480bf7d247242b8974c842a4e7c6bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:26.533313 26829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bea311fb784f60bb9970ace0504cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:26.538861 26934 tablet_bootstrap.cc:492] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319: Bootstrap starting.
I20260807 11:29:26.539883 26931 tablet_bootstrap.cc:492] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2: Bootstrap starting.
I20260807 11:29:26.541060 26934 tablet_bootstrap.cc:654] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:26.541857 26931 tablet_bootstrap.cc:654] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:26.544080 26930 tablet_bootstrap.cc:492] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf: Bootstrap starting.
I20260807 11:29:26.545017 26960 tablet_replica.cc:330] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: stopping tablet replica
I20260807 11:29:26.545532 26960 raft_consensus.cc:2229] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:29:26.545967 26930 tablet_bootstrap.cc:654] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:26.546259 26960 raft_consensus.cc:2258] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:26.547441 26957 tablet_replica.cc:330] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2: stopping tablet replica
I20260807 11:29:26.547506 26931 tablet_bootstrap.cc:492] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2: No bootstrap required, opened a new log
I20260807 11:29:26.547638 26931 ts_tablet_manager.cc:1397] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260807 11:29:26.548002 26957 raft_consensus.cc:2229] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:26.548250 26934 tablet_bootstrap.cc:492] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319: No bootstrap required, opened a new log
I20260807 11:29:26.548425 26934 ts_tablet_manager.cc:1397] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260807 11:29:26.548532 26931 raft_consensus.cc:348] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.548777 26931 raft_consensus.cc:374] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:26.548884 26931 raft_consensus.cc:729] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76096aec61c24e2fb51ae2d18af026c2, State: Initialized, Role: FOLLOWER
I20260807 11:29:26.549122 26931 consensus_queue.cc:260] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.549436 26934 raft_consensus.cc:348] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.549726 26934 raft_consensus.cc:374] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:26.549842 26934 raft_consensus.cc:729] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7336962baf4eefa9b4dceb43484319, State: Initialized, Role: FOLLOWER
I20260807 11:29:26.549849 26957 raft_consensus.cc:2258] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:26.550505 26931 ts_tablet_manager.cc:1428] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 11:29:26.550745 26934 consensus_queue.cc:260] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.552942 26934 ts_tablet_manager.cc:1428] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260807 11:29:26.554711 26959 ts_tablet_manager.cc:1905] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:26.556828 26930 tablet_bootstrap.cc:492] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf: No bootstrap required, opened a new log
I20260807 11:29:26.557015 26930 ts_tablet_manager.cc:1397] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260807 11:29:26.557963 26930 raft_consensus.cc:348] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.558231 26930 raft_consensus.cc:374] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:26.558351 26930 raft_consensus.cc:729] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20484d6c9fbc47caba26c1f2b994eeaf, State: Initialized, Role: FOLLOWER
I20260807 11:29:26.558679 26930 consensus_queue.cc:260] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.559357 26930 ts_tablet_manager.cc:1428] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260807 11:29:26.562188 26960 ts_tablet_manager.cc:1905] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:26.563549 26957 ts_tablet_manager.cc:1905] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:26.564265 26959 ts_tablet_manager.cc:1918] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:29:26.564451 26959 log.cc:1196] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/52c4b6a97c564feaa5dbd73f5ad428f2
I20260807 11:29:26.565022 26959 ts_tablet_manager.cc:1939] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 9e7336962baf4eefa9b4dceb43484319: Deleting consensus metadata
I20260807 11:29:26.567095 26310 catalog_manager.cc:4841] TS 9e7336962baf4eefa9b4dceb43484319 (127.24.103.66:39455): tablet 52c4b6a97c564feaa5dbd73f5ad428f2 (table test-table [id=ee6ae4bf8cf546ad9b2564367efb9197]) successfully deleted
I20260807 11:29:26.570843 26960 ts_tablet_manager.cc:1918] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:29:26.571060 26960 log.cc:1196] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/52c4b6a97c564feaa5dbd73f5ad428f2
I20260807 11:29:26.571590 26960 ts_tablet_manager.cc:1939] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 20484d6c9fbc47caba26c1f2b994eeaf: Deleting consensus metadata
I20260807 11:29:26.571800 26957 ts_tablet_manager.cc:1918] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:29:26.571972 26957 log.cc:1196] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/52c4b6a97c564feaa5dbd73f5ad428f2
I20260807 11:29:26.572540 26957 ts_tablet_manager.cc:1939] T 52c4b6a97c564feaa5dbd73f5ad428f2 P 76096aec61c24e2fb51ae2d18af026c2: Deleting consensus metadata
I20260807 11:29:26.573328 26314 catalog_manager.cc:4841] TS 20484d6c9fbc47caba26c1f2b994eeaf (127.24.103.67:46609): tablet 52c4b6a97c564feaa5dbd73f5ad428f2 (table test-table [id=ee6ae4bf8cf546ad9b2564367efb9197]) successfully deleted
I20260807 11:29:26.574322 26310 catalog_manager.cc:4841] TS 76096aec61c24e2fb51ae2d18af026c2 (127.24.103.65:35305): tablet 52c4b6a97c564feaa5dbd73f5ad428f2 (table test-table [id=ee6ae4bf8cf546ad9b2564367efb9197]) successfully deleted
I20260807 11:29:26.605180 26941 raft_consensus.cc:482] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:29:26.605459 26941 raft_consensus.cc:504] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.606316 26941 leader_election.cc:288] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76096aec61c24e2fb51ae2d18af026c2 (127.24.103.65:35305), 9e7336962baf4eefa9b4dceb43484319 (127.24.103.66:39455)
I20260807 11:29:26.607080 26519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bea311fb784f60bb9970ace0504cca" candidate_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76096aec61c24e2fb51ae2d18af026c2" is_pre_election: true
I20260807 11:29:26.607215 26685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bea311fb784f60bb9970ace0504cca" candidate_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e7336962baf4eefa9b4dceb43484319" is_pre_election: true
I20260807 11:29:26.607425 26519 raft_consensus.cc:2454] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20484d6c9fbc47caba26c1f2b994eeaf in term 0.
I20260807 11:29:26.607501 26685 raft_consensus.cc:2454] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20484d6c9fbc47caba26c1f2b994eeaf in term 0.
I20260807 11:29:26.607959 26766 leader_election.cc:302] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20484d6c9fbc47caba26c1f2b994eeaf, 76096aec61c24e2fb51ae2d18af026c2; no voters:
I20260807 11:29:26.608297 26941 raft_consensus.cc:2781] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:29:26.608422 26941 raft_consensus.cc:482] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:29:26.608552 26941 raft_consensus.cc:3037] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:26.610853 26941 raft_consensus.cc:504] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.611590 26941 leader_election.cc:288] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [CANDIDATE]: Term 1 election: Requested vote from peers 76096aec61c24e2fb51ae2d18af026c2 (127.24.103.65:35305), 9e7336962baf4eefa9b4dceb43484319 (127.24.103.66:39455)
I20260807 11:29:26.612139 26519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bea311fb784f60bb9970ace0504cca" candidate_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76096aec61c24e2fb51ae2d18af026c2"
I20260807 11:29:26.612226 26685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bea311fb784f60bb9970ace0504cca" candidate_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e7336962baf4eefa9b4dceb43484319"
I20260807 11:29:26.612376 26519 raft_consensus.cc:3037] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:26.612450 26685 raft_consensus.cc:3037] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:26.615177 26519 raft_consensus.cc:2454] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20484d6c9fbc47caba26c1f2b994eeaf in term 1.
I20260807 11:29:26.615692 26766 leader_election.cc:302] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20484d6c9fbc47caba26c1f2b994eeaf, 76096aec61c24e2fb51ae2d18af026c2; no voters:
I20260807 11:29:26.615839 26685 raft_consensus.cc:2454] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20484d6c9fbc47caba26c1f2b994eeaf in term 1.
I20260807 11:29:26.616080 26941 raft_consensus.cc:2781] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:26.616356 26941 raft_consensus.cc:686] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 LEADER]: Becoming Leader. State: Replica: 20484d6c9fbc47caba26c1f2b994eeaf, State: Running, Role: LEADER
I20260807 11:29:26.616685 26941 consensus_queue.cc:237] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.620222 26330 catalog_manager.cc:5495] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 20484d6c9fbc47caba26c1f2b994eeaf (127.24.103.67). New cstate: current_term: 1 leader_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } health_report { overall_health: HEALTHY } } }
Aug 07 11:29:26 dist-test-slave-wl2h krb5kdc[26272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102164, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:26.681733 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.665956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33723 (local address 127.0.0.1:55728)
0807 11:29:26.666576 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.666597 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.666746 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.667176 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.667179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.667967 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:26.668292 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.668302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.669868 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.669872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.670866 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.670877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.672083 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.672089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.672223 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.672974 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:26.672991 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:26.675348 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:26.679150 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:26.679163 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:26.679170 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:26.679646 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:26.680049 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:26.680052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:26.680054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:26.680192 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:26.680823 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:26.680829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:26.680980 (+ 151us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.681143 (+ 163us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.681384 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":349,"threads_started":1}
I20260807 11:29:26.682041 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.665858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.24.103.126:33723)
0807 11:29:26.666043 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.666048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.666714 (+ 666us) server_negotiation.cc:407] Connection header received
0807 11:29:26.666868 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.666876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.666956 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.667109 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:26.668459 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.669695 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.671044 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.671808 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.671943 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.675549 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:26.675557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:26.675563 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:26.675610 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:26.678952 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:26.679768 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:26.679774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:26.679778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:26.679932 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:26.680295 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:26.680301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:26.680304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:26.680650 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:26.680794 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.681272 (+ 478us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.681574 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":60}
I20260807 11:29:26.687678 26330 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:55728:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 11:29:26.688794 26330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:29:26.696355 26658 tablet_service.cc:1467] Processing CreateTablet for tablet e33f072adf8c4bdbb241b78e7e839c55 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e9acdf381624f20a718144f5ebbb99e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:29:26.696355 26492 tablet_service.cc:1467] Processing CreateTablet for tablet e33f072adf8c4bdbb241b78e7e839c55 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e9acdf381624f20a718144f5ebbb99e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:29:26.696714 26830 tablet_service.cc:1467] Processing CreateTablet for tablet e33f072adf8c4bdbb241b78e7e839c55 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e9acdf381624f20a718144f5ebbb99e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:29:26.696982 26658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33f072adf8c4bdbb241b78e7e839c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:26.696982 26492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33f072adf8c4bdbb241b78e7e839c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:26.697320 26830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33f072adf8c4bdbb241b78e7e839c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:26.705202 26930 tablet_bootstrap.cc:492] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf: Bootstrap starting.
I20260807 11:29:26.705196 26931 tablet_bootstrap.cc:492] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2: Bootstrap starting.
I20260807 11:29:26.705276 26934 tablet_bootstrap.cc:492] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: Bootstrap starting.
I20260807 11:29:26.707121 26931 tablet_bootstrap.cc:654] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:26.707127 26934 tablet_bootstrap.cc:654] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:26.707275 26930 tablet_bootstrap.cc:654] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:26.715711 26934 tablet_bootstrap.cc:492] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: No bootstrap required, opened a new log
I20260807 11:29:26.716023 26934 ts_tablet_manager.cc:1397] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260807 11:29:26.716171 26930 tablet_bootstrap.cc:492] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf: No bootstrap required, opened a new log
I20260807 11:29:26.716363 26930 ts_tablet_manager.cc:1397] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260807 11:29:26.717191 26930 raft_consensus.cc:348] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.717461 26930 raft_consensus.cc:374] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:26.717547 26930 raft_consensus.cc:729] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20484d6c9fbc47caba26c1f2b994eeaf, State: Initialized, Role: FOLLOWER
I20260807 11:29:26.717294 26934 raft_consensus.cc:348] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.717829 26930 consensus_queue.cc:260] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.718210 26934 raft_consensus.cc:374] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:26.718350 26934 raft_consensus.cc:729] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7336962baf4eefa9b4dceb43484319, State: Initialized, Role: FOLLOWER
I20260807 11:29:26.718340 26941 tablet_replica.cc:439] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } }
I20260807 11:29:26.718613 26941 tablet_replica.cc:442] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:26.718850 26934 consensus_queue.cc:260] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.719578 26939 tablet_replica.cc:439] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } }
I20260807 11:29:26.720027 26939 tablet_replica.cc:442] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:26.721679 26939 raft_consensus.cc:482] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:29:26.721866 26939 raft_consensus.cc:504] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.723462 26930 ts_tablet_manager.cc:1428] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260807 11:29:26.726912 26939 leader_election.cc:288] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76096aec61c24e2fb51ae2d18af026c2 (127.24.103.65:35305), 20484d6c9fbc47caba26c1f2b994eeaf (127.24.103.67:46609)
I20260807 11:29:26.732264 26934 ts_tablet_manager.cc:1428] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20260807 11:29:26.735317 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.725496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:52389 (local address 127.24.103.67:46609)
0807 11:29:26.725679 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.725685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.727458 (+ 1773us) server_negotiation.cc:407] Connection header received
0807 11:29:26.727726 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.727732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.727805 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.727866 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:26.728721 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.729951 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.732689 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.734019 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.734162 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.734234 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.734818 (+ 584us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.734975 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 11:29:26.736110 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.724700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46609 (local address 127.24.103.66:52389)
0807 11:29:26.727351 (+ 2651us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.727377 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.727569 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.728087 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.728091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.728119 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:26.728525 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.728537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.730144 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.730152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.732492 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.732508 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.734432 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.734438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.734625 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.734664 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.735598 (+ 934us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.735752 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1707,"threads_started":1}
I20260807 11:29:26.740334 26854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33f072adf8c4bdbb241b78e7e839c55" candidate_uuid: "9e7336962baf4eefa9b4dceb43484319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" is_pre_election: true
I20260807 11:29:26.740800 26854 raft_consensus.cc:2454] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e7336962baf4eefa9b4dceb43484319 in term 0.
I20260807 11:29:26.741617 26931 tablet_bootstrap.cc:492] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2: No bootstrap required, opened a new log
I20260807 11:29:26.741840 26931 ts_tablet_manager.cc:1397] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.017s
I20260807 11:29:26.742740 26603 leader_election.cc:302] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20484d6c9fbc47caba26c1f2b994eeaf, 9e7336962baf4eefa9b4dceb43484319; no voters:
I20260807 11:29:26.742708 26931 raft_consensus.cc:348] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.742956 26931 raft_consensus.cc:374] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:26.743079 26931 raft_consensus.cc:729] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76096aec61c24e2fb51ae2d18af026c2, State: Initialized, Role: FOLLOWER
I20260807 11:29:26.743217 26939 raft_consensus.cc:2781] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:29:26.743369 26939 raft_consensus.cc:482] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:29:26.743343 26931 consensus_queue.cc:260] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.743510 26939 raft_consensus.cc:3037] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:26.744549 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.725904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35305 (local address 127.24.103.66:48107)
0807 11:29:26.736336 (+ 10432us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.736356 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.736545 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.737102 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.737107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.737134 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:26.737305 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.737315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.738940 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.738945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.740877 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.740893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.743693 (+ 2800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.743699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.743930 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.743966 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.744083 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.744232 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10299,"thread_start_us":1337,"threads_started":1}
I20260807 11:29:26.744592 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.726517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:48107 (local address 127.24.103.65:35305)
0807 11:29:26.726749 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.726758 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.736441 (+ 9683us) server_negotiation.cc:407] Connection header received
0807 11:29:26.736736 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.736742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.736825 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.736890 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:26.737511 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.738716 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.741118 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.742197 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.742342 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.742396 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.744123 (+ 1727us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.744290 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:29:26.744973 26937 tablet_replica.cc:439] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } }
I20260807 11:29:26.745404 26937 tablet_replica.cc:442] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:26.745359 26519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33f072adf8c4bdbb241b78e7e839c55" candidate_uuid: "9e7336962baf4eefa9b4dceb43484319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76096aec61c24e2fb51ae2d18af026c2" is_pre_election: true
I20260807 11:29:26.745622 26519 raft_consensus.cc:2454] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e7336962baf4eefa9b4dceb43484319 in term 0.
I20260807 11:29:26.746829 26931 ts_tablet_manager.cc:1428] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260807 11:29:26.748628 26939 raft_consensus.cc:504] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.749325 26939 leader_election.cc:288] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [CANDIDATE]: Term 1 election: Requested vote from peers 76096aec61c24e2fb51ae2d18af026c2 (127.24.103.65:35305), 20484d6c9fbc47caba26c1f2b994eeaf (127.24.103.67:46609)
I20260807 11:29:26.749753 26519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33f072adf8c4bdbb241b78e7e839c55" candidate_uuid: "9e7336962baf4eefa9b4dceb43484319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76096aec61c24e2fb51ae2d18af026c2"
I20260807 11:29:26.749991 26519 raft_consensus.cc:3037] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:26.750458 26854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33f072adf8c4bdbb241b78e7e839c55" candidate_uuid: "9e7336962baf4eefa9b4dceb43484319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20484d6c9fbc47caba26c1f2b994eeaf"
I20260807 11:29:26.750684 26854 raft_consensus.cc:3037] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:26.752164 26519 raft_consensus.cc:2454] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e7336962baf4eefa9b4dceb43484319 in term 1.
I20260807 11:29:26.752588 26599 leader_election.cc:302] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76096aec61c24e2fb51ae2d18af026c2, 9e7336962baf4eefa9b4dceb43484319; no voters:
I20260807 11:29:26.752905 26939 raft_consensus.cc:2781] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:26.754154 26854 raft_consensus.cc:2454] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e7336962baf4eefa9b4dceb43484319 in term 1.
I20260807 11:29:26.753500 26939 raft_consensus.cc:686] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [term 1 LEADER]: Becoming Leader. State: Replica: 9e7336962baf4eefa9b4dceb43484319, State: Running, Role: LEADER
I20260807 11:29:26.755091 26939 consensus_queue.cc:237] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } }
I20260807 11:29:26.757618 26976 tablet_replica.cc:439] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: TxnStatusTablet state changed. Reason: New leader 9e7336962baf4eefa9b4dceb43484319. Latest consensus state: current_term: 1 leader_uuid: "9e7336962baf4eefa9b4dceb43484319" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } }
I20260807 11:29:26.758023 26976 tablet_replica.cc:442] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:29:26.758874 26978 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 11:29:26.759137 26978 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 11:29:26.759142 26330 catalog_manager.cc:5495] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e7336962baf4eefa9b4dceb43484319 (127.24.103.66). New cstate: current_term: 1 leader_uuid: "9e7336962baf4eefa9b4dceb43484319" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } health_report { overall_health: UNKNOWN } } }
W20260807 11:29:26.776928 26573 tablet.cc:2369] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:29:26.780565 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.773150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.24.103.66:39455)
0807 11:29:26.773454 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.773460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.773489 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:29:26.773611 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.773616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.773727 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.773881 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:26.774344 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.775604 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.777676 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.778340 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.778477 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.779117 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:26.779819 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:26.779977 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.780149 (+ 172us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.780241 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260807 11:29:26.780630 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.772942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:39455 (local address 127.0.0.1:50402)
0807 11:29:26.773191 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.773214 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.773354 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.773924 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.773928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.773941 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:29:26.774151 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.774164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.775786 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.775793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.777511 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.777528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.778529 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.778536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.778763 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.778850 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:29:26.779969 (+ 1119us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:29:26.780004 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.780128 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.780300 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 11:29:26.785717 26854 raft_consensus.cc:1261] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 FOLLOWER]: Refusing update from remote peer 9e7336962baf4eefa9b4dceb43484319: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:26.785727 26519 raft_consensus.cc:1261] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2 [term 1 FOLLOWER]: Refusing update from remote peer 9e7336962baf4eefa9b4dceb43484319: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:26.786590 26939 consensus_queue.cc:1035] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:26.786990 26976 consensus_queue.cc:1035] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:26.798440 26941 tablet_replica.cc:439] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf: TxnStatusTablet state changed. Reason: New leader 9e7336962baf4eefa9b4dceb43484319. Latest consensus state: current_term: 1 leader_uuid: "9e7336962baf4eefa9b4dceb43484319" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } }
I20260807 11:29:26.798925 26941 tablet_replica.cc:442] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:26.800307 26937 tablet_replica.cc:439] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2: TxnStatusTablet state changed. Reason: New leader 9e7336962baf4eefa9b4dceb43484319. Latest consensus state: current_term: 1 leader_uuid: "9e7336962baf4eefa9b4dceb43484319" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } }
I20260807 11:29:26.800635 26937 tablet_replica.cc:442] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:26.805943 26937 tablet_replica.cc:439] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e7336962baf4eefa9b4dceb43484319" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } }
I20260807 11:29:26.806254 26937 tablet_replica.cc:442] T e33f072adf8c4bdbb241b78e7e839c55 P 76096aec61c24e2fb51ae2d18af026c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:26.807806 26941 tablet_replica.cc:439] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e7336962baf4eefa9b4dceb43484319" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } }
I20260807 11:29:26.808130 26941 tablet_replica.cc:442] T e33f072adf8c4bdbb241b78e7e839c55 P 20484d6c9fbc47caba26c1f2b994eeaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:26.811757 26986 tablet_replica.cc:439] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e7336962baf4eefa9b4dceb43484319" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } }
I20260807 11:29:26.812232 26986 tablet_replica.cc:442] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:29:26.812482 26986 tablet_replica.cc:439] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e7336962baf4eefa9b4dceb43484319" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 } } peers { permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 } } peers { permanent_uuid: "20484d6c9fbc47caba26c1f2b994eeaf" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46609 } } }
I20260807 11:29:26.812693 26986 tablet_replica.cc:442] T e33f072adf8c4bdbb241b78e7e839c55 P 9e7336962baf4eefa9b4dceb43484319: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:29:26.841084 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.831266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:41693 (local address 127.24.103.66:39455)
0807 11:29:26.831557 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.831567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.835743 (+ 4176us) server_negotiation.cc:407] Connection header received
0807 11:29:26.835991 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.835995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.836072 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.836184 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:26.836755 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.837532 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.839081 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.839676 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.839840 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.840309 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:26.840415 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:26.840512 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.840675 (+ 163us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.840761 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 11:29:26.841315 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.831136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:39455 (local address 127.24.103.67:41693)
0807 11:29:26.835617 (+ 4481us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.835638 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.835833 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.836204 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.836208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.836220 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:29:26.836587 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.836596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.837696 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.837702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.838902 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.838916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.839916 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.839923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.840060 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.840116 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:29:26.840519 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:29:26.840532 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.840657 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.840814 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4303,"thread_start_us":4144,"threads_started":1}
I20260807 11:29:26.854512 26518 raft_consensus.cc:1261] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2 [term 1 FOLLOWER]: Refusing update from remote peer 20484d6c9fbc47caba26c1f2b994eeaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:26.854583 26685 raft_consensus.cc:1261] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319 [term 1 FOLLOWER]: Refusing update from remote peer 20484d6c9fbc47caba26c1f2b994eeaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:26.855343 26954 consensus_queue.cc:1035] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "76096aec61c24e2fb51ae2d18af026c2" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:26.855985 26954 consensus_queue.cc:1035] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e7336962baf4eefa9b4dceb43484319" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:26.866241 26945 mvcc.cc:205] Tried to move back new op lower bound from 7315874475429433344 to 7315874474465755136. Current Snapshot: MvccSnapshot[applied={T|T < 7315874475429433344}]
I20260807 11:29:26.909940 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.900492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:45899 (local address 127.24.103.67:46609)
0807 11:29:26.900741 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.900748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.901640 (+ 892us) server_negotiation.cc:407] Connection header received
0807 11:29:26.901991 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.901997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.902098 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.902187 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:26.903568 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.904485 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.907663 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.908276 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.908430 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.909087 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:26.909220 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:26.909332 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.909486 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 11:29:26.909571 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 11:29:26.910032 26994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.900491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46609 (local address 127.24.103.66:45899)
0807 11:29:26.901522 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:26.901549 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:26.901791 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:26.902410 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:26.902415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:26.902427 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:29:26.903343 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.903357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.904628 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.904672 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.907487 (+ 2815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:26.907503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.908622 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.908627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:26.908811 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.908895 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:29:26.909336 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:29:26.909351 (+ 15us) client_negotiation.cc:769] Sending connection context
0807 11:29:26.909455 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 11:29:26.909600 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":2028,"threads_started":1}
I20260807 11:29:26.953287 26331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:26.953600 26331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:26.957111 26331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0fb1a7fbf5a4bd9b1c3c61fc363805e: Sending DeleteTablet for 3 replicas of tablet b3bea311fb784f60bb9970ace0504cca
I20260807 11:29:26.958223 26492 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bea311fb784f60bb9970ace0504cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:26 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:33274
I20260807 11:29:26.958223 26830 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bea311fb784f60bb9970ace0504cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:26 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:50898
I20260807 11:29:26.958526 26960 tablet_replica.cc:330] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf: stopping tablet replica
I20260807 11:29:26.958566 26957 tablet_replica.cc:330] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2: stopping tablet replica
I20260807 11:29:26.958719 26960 raft_consensus.cc:2229] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:29:26.958808 26957 raft_consensus.cc:2229] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:26.959033 26957 raft_consensus.cc:2258] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:26.959137 26960 raft_consensus.cc:2258] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:26.959122 26658 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bea311fb784f60bb9970ace0504cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:26 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:50392
I20260807 11:29:26.960351 26960 ts_tablet_manager.cc:1905] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:26.960405 26957 ts_tablet_manager.cc:1905] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:26.962945 26959 tablet_replica.cc:330] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319: stopping tablet replica
I20260807 11:29:26.963219 26959 raft_consensus.cc:2229] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:26.963451 26959 raft_consensus.cc:2258] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:26.964824 26959 ts_tablet_manager.cc:1905] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:26.968829 26960 ts_tablet_manager.cc:1918] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:29:26.969022 26960 log.cc:1196] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/b3bea311fb784f60bb9970ace0504cca
I20260807 11:29:26.969506 26960 ts_tablet_manager.cc:1939] T b3bea311fb784f60bb9970ace0504cca P 20484d6c9fbc47caba26c1f2b994eeaf: Deleting consensus metadata
I20260807 11:29:26.970979 26314 catalog_manager.cc:4841] TS 20484d6c9fbc47caba26c1f2b994eeaf (127.24.103.67:46609): tablet b3bea311fb784f60bb9970ace0504cca (table test-table [id=e480bf7d247242b8974c842a4e7c6bcd]) successfully deleted
I20260807 11:29:26.974272 26959 ts_tablet_manager.cc:1918] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:29:26.975291 26957 ts_tablet_manager.cc:1918] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:29:26.975474 26957 log.cc:1196] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/b3bea311fb784f60bb9970ace0504cca
I20260807 11:29:26.976012 26957 ts_tablet_manager.cc:1939] T b3bea311fb784f60bb9970ace0504cca P 76096aec61c24e2fb51ae2d18af026c2: Deleting consensus metadata
I20260807 11:29:26.976284 26959 log.cc:1196] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/b3bea311fb784f60bb9970ace0504cca
I20260807 11:29:26.976979 26959 ts_tablet_manager.cc:1939] T b3bea311fb784f60bb9970ace0504cca P 9e7336962baf4eefa9b4dceb43484319: Deleting consensus metadata
I20260807 11:29:26.977311 26310 catalog_manager.cc:4841] TS 76096aec61c24e2fb51ae2d18af026c2 (127.24.103.65:35305): tablet b3bea311fb784f60bb9970ace0504cca (table test-table [id=e480bf7d247242b8974c842a4e7c6bcd]) successfully deleted
I20260807 11:29:26.980510 26310 catalog_manager.cc:4841] TS 9e7336962baf4eefa9b4dceb43484319 (127.24.103.66:39455): tablet b3bea311fb784f60bb9970ace0504cca (table test-table [id=e480bf7d247242b8974c842a4e7c6bcd]) successfully deleted
Aug 07 11:29:26 dist-test-slave-wl2h krb5kdc[26272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102166, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:29:27.000680 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:26.970565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.24.103.65:35305)
0807 11:29:26.970778 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:26.970784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:26.972435 (+ 1651us) server_negotiation.cc:407] Connection header received
0807 11:29:26.972833 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:26.972840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:26.972961 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:26.973064 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:26.975694 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.977070 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.979380 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:26.980098 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:26.980238 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:26.987303 (+ 7065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:26.987330 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:26.987352 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:26.987415 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:26.996663 (+ 9248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:26.997776 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:26.997789 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:26.997793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:26.998007 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:26.998486 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:26.998493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:26.998496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:26.999069 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:26.999426 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:26.999965 (+ 539us) server_negotiation.cc:299] Negotiation successful
0807 11:29:27.000252 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 11:29:27.001632 26566 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33288
I20260807 11:29:27.030989 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:27.006156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.24.103.126:33723)
0807 11:29:27.006392 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:27.006399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:27.011725 (+ 5326us) server_negotiation.cc:407] Connection header received
0807 11:29:27.011959 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:27.011965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:27.012077 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:27.012210 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:27.014175 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:27.015375 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:27.017332 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:27.018061 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:27.018215 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:27.021662 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:27.021670 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:27.021675 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:27.021742 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:27.026077 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:27.027628 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:27.027634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:27.027637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:27.027829 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:27.028336 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:27.028344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:27.028350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:27.028828 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:27.029080 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:27.030329 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0807 11:29:27.030586 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260807 11:29:27.032234 26331 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55734
I20260807 11:29:27.034370 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 26412
I20260807 11:29:27.065900 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:27.035807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.24.103.65:35305)
0807 11:29:27.036020 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:27.036026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:27.047742 (+ 11716us) server_negotiation.cc:407] Connection header received
0807 11:29:27.047897 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:27.047903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:27.048018 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:27.048165 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:27.049873 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:27.051086 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:27.053045 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:27.053773 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:27.053910 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:27.057450 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:27.057456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:27.057461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:27.057502 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:27.061584 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:27.062517 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:27.062524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:27.062527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:27.062677 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:27.063288 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:27.063294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:27.063297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:27.063690 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:27.064072 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:27.065255 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0807 11:29:27.065493 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 11:29:27.067214 26566 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33294
W20260807 11:29:27.067911 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 26412: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:29:27.068111 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 26412
I20260807 11:29:27.097851 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 26576
Aug 07 11:29:27 dist-test-slave-wl2h krb5kdc[26272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102166, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:29:27.118948 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:27.098791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.24.103.66:39455)
0807 11:29:27.099061 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:27.099067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:27.099313 (+ 246us) server_negotiation.cc:407] Connection header received
0807 11:29:27.099510 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:27.099515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:27.099593 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:27.099699 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:27.101403 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:27.102564 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:27.104573 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:27.105232 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:27.105367 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:27.111160 (+ 5793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:27.111175 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:27.111193 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:27.111251 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:27.115447 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:27.116373 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:27.116383 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:27.116388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:27.116547 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:27.117002 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:27.117007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:27.117011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:27.117489 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:27.117859 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:27.118157 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 11:29:27.118391 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260807 11:29:27.119812 26735 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50412
W20260807 11:29:27.120615 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 26576: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:29:27.120816 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 26576
I20260807 11:29:27.149873 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 26744
Aug 07 11:29:27 dist-test-slave-wl2h krb5kdc[26272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102166, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:29:27.168679 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:27.150791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.24.103.67:46609)
0807 11:29:27.150994 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:27.151001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:27.151065 (+ 64us) server_negotiation.cc:407] Connection header received
0807 11:29:27.151182 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:27.151190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:27.151288 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:27.151397 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:27.153135 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:27.154349 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:27.155771 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:27.156474 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:27.156594 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:27.161207 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:27.161220 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:27.161239 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:27.161276 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:27.165030 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:27.165889 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:27.165897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:27.165900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:27.166129 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:27.166580 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:27.166583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:27.166586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:27.167213 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:27.167570 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:27.167938 (+ 368us) server_negotiation.cc:299] Negotiation successful
0807 11:29:27.168331 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 11:29:27.169646 26902 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50926
W20260807 11:29:27.170604 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 26744: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:29:27.170781 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 26744
I20260807 11:29:27.193200 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 26290
I20260807 11:29:27.194468 26376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:27.313722 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 26290
2026-08-07T11:29:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4840 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:29:27 dist-test-slave-wl2h krb5kdc[27005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:29:27 dist-test-slave-wl2h krb5kdc[27005](info): set up 2 sockets
Aug 07 11:29:27 dist-test-slave-wl2h krb5kdc[27005](info): commencing operation
krb5kdc: starting...
W20260807 11:29:28.511981 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:29:28 dist-test-slave-wl2h krb5kdc[27005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:29:28.747987 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39055
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:39055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:29.084645 27022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:29.084991 27022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:29.085161 27022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:29.092341 27022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:29.092497 27022 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 11:29:29.092594 27022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:29.092707 27022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:29.092801 27022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:29.092913 27022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:29:29.102981 27022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:39055
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:29.104463 27022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:29.105860 27022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:29.118312 27028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:29.118376 27030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:29.118353 27027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:29.119517 27022 server_base.cc:1034] running on GCE node
I20260807 11:29:29.120273 27022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:29.121697 27022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:29.122957 27022 hybrid_clock.cc:648] HybridClock initialized: now 1786102169122893 us; error 34 us; skew 500 ppm
I20260807 11:29:29.123324 27022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:29 dist-test-slave-wl2h krb5kdc[27005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102169, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:29.126637 27022 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:29:29.128284 27022 webserver.cc:460] Webserver started at http://127.24.103.126:41165/ using document root <none> and password file <none>
I20260807 11:29:29.128994 27022 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:29.129134 27022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:29.129465 27022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:29.132553 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "e6cdd1343b864a8fa79d6ccfb74fe34b"
format_stamp: "Formatted at 2026-08-07 11:29:29 on dist-test-slave-wl2h"
server_key: "9b567c3aef8d3ab61d51c92f4f907f78"
server_key_iv: "cd0e1c60e997db89ca6da13eeea876ce"
server_key_version: "encryptionkey@0"
I20260807 11:29:29.133347 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "e6cdd1343b864a8fa79d6ccfb74fe34b"
format_stamp: "Formatted at 2026-08-07 11:29:29 on dist-test-slave-wl2h"
server_key: "9b567c3aef8d3ab61d51c92f4f907f78"
server_key_iv: "cd0e1c60e997db89ca6da13eeea876ce"
server_key_version: "encryptionkey@0"
I20260807 11:29:29.138667 27022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 11:29:29.142364 27039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:29.143898 27022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:29.144088 27022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "e6cdd1343b864a8fa79d6ccfb74fe34b"
format_stamp: "Formatted at 2026-08-07 11:29:29 on dist-test-slave-wl2h"
server_key: "9b567c3aef8d3ab61d51c92f4f907f78"
server_key_iv: "cd0e1c60e997db89ca6da13eeea876ce"
server_key_version: "encryptionkey@0"
I20260807 11:29:29.144328 27022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:29.161791 27022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:29.166531 27022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:29.166832 27022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:29.191280 27022 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:39055
I20260807 11:29:29.191284 27112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:39055 every 8 connection(s)
I20260807 11:29:29.192761 27022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:29.196877 27113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:29.201687 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 27022
I20260807 11:29:29.201961 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:29.202907 24989 external_mini_cluster.cc:1428] Setting key b17c5610c5a7109c377be30565ba5552
I20260807 11:29:29.210093 27113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b: Bootstrap starting.
I20260807 11:29:29.213750 27113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:29.214921 27113 log.cc:824] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:29.218227 27113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b: No bootstrap required, opened a new log
Aug 07 11:29:29 dist-test-slave-wl2h krb5kdc[27005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:29.228348 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:29.205414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.24.103.126:39055)
0807 11:29:29.206301 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:29.206313 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:29.206355 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:29:29.208649 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:29.208691 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:29.209100 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:29.209481 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:29.211113 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:29.212449 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:29.213770 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:29.214571 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:29.214847 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:29.219457 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:29.219467 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:29.219485 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:29.219553 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:29.223598 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:29.224278 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:29.224287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:29.224295 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:29.224514 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:29.224973 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:29.224979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:29.224982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:29.225740 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:29.226062 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:29.226401 (+ 339us) server_negotiation.cc:299] Negotiation successful
0807 11:29:29.226886 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":427,"threads_started":1}
I20260807 11:29:29.229310 27113 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdd1343b864a8fa79d6ccfb74fe34b" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39055 } }
I20260807 11:29:29.229763 27113 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:29.229915 27113 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6cdd1343b864a8fa79d6ccfb74fe34b, State: Initialized, Role: FOLLOWER
I20260807 11:29:29.230480 27113 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdd1343b864a8fa79d6ccfb74fe34b" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39055 } }
I20260807 11:29:29.230752 27113 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:29.230908 27113 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:29.231120 27113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:29.233485 27113 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdd1343b864a8fa79d6ccfb74fe34b" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39055 } }
I20260807 11:29:29.233975 27113 leader_election.cc:302] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6cdd1343b864a8fa79d6ccfb74fe34b; no voters:
I20260807 11:29:29.234589 27113 leader_election.cc:288] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:29.234786 27120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:29.236153 27120 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [term 1 LEADER]: Becoming Leader. State: Replica: e6cdd1343b864a8fa79d6ccfb74fe34b, State: Running, Role: LEADER
I20260807 11:29:29.236847 27113 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:29.236811 27120 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdd1343b864a8fa79d6ccfb74fe34b" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39055 } }
I20260807 11:29:29.240285 27121 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6cdd1343b864a8fa79d6ccfb74fe34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdd1343b864a8fa79d6ccfb74fe34b" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39055 } } }
I20260807 11:29:29.240221 27122 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6cdd1343b864a8fa79d6ccfb74fe34b. Latest consensus state: current_term: 1 leader_uuid: "e6cdd1343b864a8fa79d6ccfb74fe34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cdd1343b864a8fa79d6ccfb74fe34b" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39055 } } }
I20260807 11:29:29.240707 27121 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:29.240708 27122 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:29.248131 27129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:29.254037 27129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:29.263455 27129 catalog_manager.cc:1347] Generated new cluster ID: 88da49ba49774a1f9eadbb8fc041c641
I20260807 11:29:29.263582 27129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:29.287442 27129 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:29.288831 27129 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:29.304785 27129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b: Generated new TSK 0
I20260807 11:29:29.305727 27129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:29:29.375698 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39055
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:29.733181 27144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:29.733482 27144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:29.733659 27144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:29.741362 27144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:29.741482 27144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:29.741676 27144 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 11:29:29.741760 27144 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:29:29.741853 27144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:29:29.752485 27144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:39055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:29.753921 27144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:29.755378 27144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:29.769256 27150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:29.769290 27151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:29.770030 27153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:29.770366 27144 server_base.cc:1034] running on GCE node
I20260807 11:29:29.771001 27144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:29.772086 27144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:29.773316 27144 hybrid_clock.cc:648] HybridClock initialized: now 1786102169773255 us; error 48 us; skew 500 ppm
I20260807 11:29:29.773675 27144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:29 dist-test-slave-wl2h krb5kdc[27005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102169, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:29.777181 27144 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:29:29.778733 27144 webserver.cc:460] Webserver started at http://127.24.103.65:42891/ using document root <none> and password file <none>
I20260807 11:29:29.779525 27144 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:29.779672 27144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:29.780018 27144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:29.783501 27144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "4837e34e7d4c4af9b0df7a291c67e581"
format_stamp: "Formatted at 2026-08-07 11:29:29 on dist-test-slave-wl2h"
server_key: "80b75775b90669b9dca387b54312068f"
server_key_iv: "2f0b4485352703387fca28a3c59cfe22"
server_key_version: "encryptionkey@0"
I20260807 11:29:29.784334 27144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "4837e34e7d4c4af9b0df7a291c67e581"
format_stamp: "Formatted at 2026-08-07 11:29:29 on dist-test-slave-wl2h"
server_key: "80b75775b90669b9dca387b54312068f"
server_key_iv: "2f0b4485352703387fca28a3c59cfe22"
server_key_version: "encryptionkey@0"
I20260807 11:29:29.789844 27144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 11:29:29.793524 27160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:29.795467 27144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:29:29.795611 27144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "4837e34e7d4c4af9b0df7a291c67e581"
format_stamp: "Formatted at 2026-08-07 11:29:29 on dist-test-slave-wl2h"
server_key: "80b75775b90669b9dca387b54312068f"
server_key_iv: "2f0b4485352703387fca28a3c59cfe22"
server_key_version: "encryptionkey@0"
I20260807 11:29:29.795789 27144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:29.825445 27144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:29.830200 27144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:29.830610 27144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:29.833753 27144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:29.833906 27144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:29.834035 27144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:29.834143 27144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:29:29 dist-test-slave-wl2h krb5kdc[27005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102169, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:29.860026 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:29.834576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:55811 (local address 127.24.103.126:39055)
0807 11:29:29.835246 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:29.835251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:29.836512 (+ 1261us) server_negotiation.cc:407] Connection header received
0807 11:29:29.838878 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:29.838884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:29.838982 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:29.839145 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:29.841297 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:29.842772 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:29.845626 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:29.846327 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:29.846474 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:29.851969 (+ 5495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:29.851974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:29.851978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:29.852026 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:29.856361 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:29.857508 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:29.857512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:29.857514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:29.857649 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:29.858136 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:29.858145 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:29.858147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:29.858420 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:29.858597 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:29.859415 (+ 818us) server_negotiation.cc:299] Negotiation successful
0807 11:29:29.859666 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":402,"threads_started":1}
I20260807 11:29:29.861677 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:29.835353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39055 (local address 127.24.103.65:55811)
0807 11:29:29.836348 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:29.836394 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:29.838589 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:29.839413 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:29.839421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:29.840181 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:29.841058 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:29.841094 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:29.843028 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:29.843037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:29.845422 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:29.845436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:29.846652 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:29.846662 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:29.846938 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:29.847903 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:29.847942 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:29.851631 (+ 3689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:29.856636 (+ 5005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:29.856647 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:29.856671 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:29.857320 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:29.857775 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:29.857780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:29.857784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:29.858003 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:29.858568 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:29.858577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:29.858946 (+ 369us) client_negotiation.cc:769] Sending connection context
0807 11:29:29.859293 (+ 347us) client_negotiation.cc:240] Negotiation successful
0807 11:29:29.859903 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":428,"threads_started":1}
I20260807 11:29:29.889155 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:29.868739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:53215 (local address 127.24.103.126:39055)
0807 11:29:29.868977 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:29.868987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:29.869879 (+ 892us) server_negotiation.cc:407] Connection header received
0807 11:29:29.870197 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:29.870205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:29.870317 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:29.870437 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:29.872386 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:29.873559 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:29.876095 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:29.876839 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:29.877026 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:29.879405 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:29.879414 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:29.879422 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:29.879479 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:29.884478 (+ 4999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:29.885982 (+ 1504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:29.885991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:29.885995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:29.886204 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:29.886957 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:29.886965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:29.886972 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:29.887468 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:29.887676 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:29.888269 (+ 593us) server_negotiation.cc:299] Negotiation successful
0807 11:29:29.888586 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 11:29:29.889431 27262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:29.868720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39055 (local address 127.24.103.65:53215)
0807 11:29:29.869735 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:29.869762 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:29.870017 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:29.870651 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:29.870655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:29.871534 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:29.872212 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:29.872232 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:29.873750 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:29.873759 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:29.875919 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:29.875936 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:29.877394 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:29.877402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:29.877559 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:29.878277 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:29.878296 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:29.879148 (+ 852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:29.884728 (+ 5580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:29.884742 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:29.884750 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:29.885760 (+ 1010us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:29.886371 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:29.886379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:29.886386 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:29.886623 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:29.887595 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:29.887599 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:29.887891 (+ 292us) client_negotiation.cc:769] Sending connection context
0807 11:29:29.888108 (+ 217us) client_negotiation.cc:240] Negotiation successful
0807 11:29:29.888616 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":547,"threads_started":1}
I20260807 11:29:29.892143 27144 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:37821
I20260807 11:29:29.892277 27301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:37821 every 8 connection(s)
I20260807 11:29:29.893620 27144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:29:29.896150 27302 heartbeater.cc:347] Connected to a master server at 127.24.103.126:39055
I20260807 11:29:29.896368 27302 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:29.897053 27302 heartbeater.cc:511] Master 127.24.103.126:39055 requested a full tablet report, sending...
I20260807 11:29:29.897526 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 27144
I20260807 11:29:29.897760 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:29:29.898563 24989 external_mini_cluster.cc:1428] Setting key aa9d7d5f932c4393f689ad9f69382ca5
I20260807 11:29:29.899534 27059 ts_manager.cc:194] Registered new tserver with Master: 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821)
I20260807 11:29:29.901533 27059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:55811
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:29:29.951246 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39055
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:30.287169 27306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:30.287469 27306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:30.287659 27306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:30.295365 27306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:30.295537 27306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:30.295701 27306 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 11:29:30.295791 27306 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:29:30.295928 27306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:29:30.307117 27306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:39055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:30.308588 27306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:30.310004 27306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:30.323832 27312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:30.323810 27313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:30.323866 27315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:30.324894 27306 server_base.cc:1034] running on GCE node
I20260807 11:29:30.325506 27306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:30.326419 27306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:30.327625 27306 hybrid_clock.cc:648] HybridClock initialized: now 1786102170327560 us; error 65 us; skew 500 ppm
I20260807 11:29:30.327960 27306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:30 dist-test-slave-wl2h krb5kdc[27005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:30.331262 27306 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:29:30.332934 27306 webserver.cc:460] Webserver started at http://127.24.103.66:45355/ using document root <none> and password file <none>
I20260807 11:29:30.333683 27306 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:30.333833 27306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:30.334163 27306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:30.337111 27306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4"
format_stamp: "Formatted at 2026-08-07 11:29:30 on dist-test-slave-wl2h"
server_key: "cac7f58855ed6f902283edd8c70d0b80"
server_key_iv: "6f8351a5ae4bce7fba9947178dc9b9f3"
server_key_version: "encryptionkey@0"
I20260807 11:29:30.337942 27306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4"
format_stamp: "Formatted at 2026-08-07 11:29:30 on dist-test-slave-wl2h"
server_key: "cac7f58855ed6f902283edd8c70d0b80"
server_key_iv: "6f8351a5ae4bce7fba9947178dc9b9f3"
server_key_version: "encryptionkey@0"
I20260807 11:29:30.343235 27306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 11:29:30.346805 27324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:30.348395 27306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:30.348574 27306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4"
format_stamp: "Formatted at 2026-08-07 11:29:30 on dist-test-slave-wl2h"
server_key: "cac7f58855ed6f902283edd8c70d0b80"
server_key_iv: "6f8351a5ae4bce7fba9947178dc9b9f3"
server_key_version: "encryptionkey@0"
I20260807 11:29:30.348783 27306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:30.362694 27306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:30.367108 27306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:30.367405 27306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:30.370352 27306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:30.370505 27306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:30.370682 27306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:30.370829 27306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 07 11:29:30 dist-test-slave-wl2h krb5kdc[27005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:30.393846 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:30.370919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:40943 (local address 127.24.103.126:39055)
0807 11:29:30.371129 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:30.371134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:30.372652 (+ 1518us) server_negotiation.cc:407] Connection header received
0807 11:29:30.374705 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:30.374711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:30.374813 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:30.374927 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:30.377185 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.378522 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.381228 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.382053 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.382195 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:30.386899 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:30.386904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:30.386907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:30.386943 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:30.389821 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:30.390694 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:30.390698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:30.390700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:30.390824 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:30.391366 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:30.391373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:30.391376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:30.391659 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:30.391845 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:30.392916 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0807 11:29:30.393249 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 11:29:30.394508 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:30.371525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39055 (local address 127.24.103.66:40943)
0807 11:29:30.372461 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:30.372515 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:30.374400 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:30.375186 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:30.375194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:30.375965 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:30.376907 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:30.376943 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.378723 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.378731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:30.381020 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:30.381041 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.382344 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.382358 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:30.382722 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:30.383679 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:30.383723 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:30.386672 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:30.389965 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:30.389974 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:30.389990 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:30.390548 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:30.390939 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:30.390945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:30.390949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:30.391236 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:30.391805 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:30.391814 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:30.392373 (+ 559us) client_negotiation.cc:769] Sending connection context
0807 11:29:30.392737 (+ 364us) client_negotiation.cc:240] Negotiation successful
0807 11:29:30.393193 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":375,"threads_started":1}
I20260807 11:29:30.417311 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:30.400730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:44879 (local address 127.24.103.126:39055)
0807 11:29:30.400936 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:30.400942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:30.401768 (+ 826us) server_negotiation.cc:407] Connection header received
0807 11:29:30.401991 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:30.401998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:30.402090 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:30.402197 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:30.403893 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.404891 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.406934 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.407666 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.407804 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:30.410661 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:30.410668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:30.410674 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:30.410732 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:30.414284 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:30.414975 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:30.414983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:30.414988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:30.415212 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:30.415713 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:30.415719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:30.415723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:30.416133 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:30.416294 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:30.416720 (+ 426us) server_negotiation.cc:299] Negotiation successful
0807 11:29:30.416947 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:29:30.417528 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:30.400860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39055 (local address 127.24.103.66:44879)
0807 11:29:30.401635 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:30.401659 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:30.401825 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:30.402366 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:30.402370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:30.403204 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:30.403643 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:30.403673 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.405081 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.405089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:30.406786 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:30.406798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.408204 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.408212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:30.408451 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:30.409150 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:30.409190 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:30.410409 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:30.414424 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:30.414430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:30.414434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:30.414847 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:30.415366 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:30.415374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:30.415382 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:30.415524 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:30.416278 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:30.416283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:30.416473 (+ 190us) client_negotiation.cc:769] Sending connection context
0807 11:29:30.416633 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 11:29:30.417011 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":376,"threads_started":1}
I20260807 11:29:30.430120 27306 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:36761
I20260807 11:29:30.430171 27467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:36761 every 8 connection(s)
I20260807 11:29:30.431470 27306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:29:30.433727 27469 heartbeater.cc:347] Connected to a master server at 127.24.103.126:39055
I20260807 11:29:30.433966 27469 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:30.434625 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 27306
I20260807 11:29:30.434679 27469 heartbeater.cc:511] Master 127.24.103.126:39055 requested a full tablet report, sending...
I20260807 11:29:30.434837 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:29:30.435433 24989 external_mini_cluster.cc:1428] Setting key e0eddfa27fc745ba08a9c7f2ed2721aa
I20260807 11:29:30.436381 27059 ts_manager.cc:194] Registered new tserver with Master: f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66:36761)
I20260807 11:29:30.437511 27059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:40943
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:29:30.488224 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39055
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:30.831846 27473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:30.832228 27473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:30.832428 27473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:30.840428 27473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:30.840587 27473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:30.840745 27473 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 11:29:30.840840 27473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:29:30.840986 27473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:29:30.851675 27473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:39055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:30.853176 27473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:30.854656 27473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:30.868897 27478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:30.868983 27481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:30.868861 27479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:30.869627 27473 server_base.cc:1034] running on GCE node
I20260807 11:29:30.870319 27473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:30.871405 27473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:30.872641 27473 hybrid_clock.cc:648] HybridClock initialized: now 1786102170872555 us; error 76 us; skew 500 ppm
I20260807 11:29:30.872967 27473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:30 dist-test-slave-wl2h krb5kdc[27005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:30.876350 27473 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:29:30.877948 27473 webserver.cc:460] Webserver started at http://127.24.103.67:34123/ using document root <none> and password file <none>
I20260807 11:29:30.878624 27473 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:30.878731 27473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:30.878999 27473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:30.882184 27473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "48be9ef638fa43e390b78079c00d0159"
format_stamp: "Formatted at 2026-08-07 11:29:30 on dist-test-slave-wl2h"
server_key: "4534103fde51ed1d65b134dcd01176f9"
server_key_iv: "f1921d7605c4740461776a4380232094"
server_key_version: "encryptionkey@0"
I20260807 11:29:30.882952 27473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "48be9ef638fa43e390b78079c00d0159"
format_stamp: "Formatted at 2026-08-07 11:29:30 on dist-test-slave-wl2h"
server_key: "4534103fde51ed1d65b134dcd01176f9"
server_key_iv: "f1921d7605c4740461776a4380232094"
server_key_version: "encryptionkey@0"
I20260807 11:29:30.888448 27473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:29:30.892642 27488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:30.894148 27473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:29:30.894340 27473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "48be9ef638fa43e390b78079c00d0159"
format_stamp: "Formatted at 2026-08-07 11:29:30 on dist-test-slave-wl2h"
server_key: "4534103fde51ed1d65b134dcd01176f9"
server_key_iv: "f1921d7605c4740461776a4380232094"
server_key_version: "encryptionkey@0"
I20260807 11:29:30.894562 27473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:30.904954 27302 heartbeater.cc:503] Master 127.24.103.126:39055 was elected leader, sending a full tablet report...
I20260807 11:29:30.919106 27473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:30.924088 27473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:30.924376 27473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:30.927258 27473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:30.927366 27473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:30.927533 27473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:30.927630 27473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:29:30 dist-test-slave-wl2h krb5kdc[27005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:30.954154 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:30.928781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39055 (local address 127.24.103.67:45585)
0807 11:29:30.929760 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:30.929812 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:30.931963 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:30.932987 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:30.933003 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:30.933847 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:30.934919 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:30.934956 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.936543 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.936559 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:30.939090 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:30.939110 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.940037 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.940047 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:30.940429 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:30.941357 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:30.941409 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:30.944946 (+ 3537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:30.948844 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:30.948857 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:30.948885 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:30.949406 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:30.949883 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:30.949889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:30.949896 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:30.950234 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:30.951078 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:30.951086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:30.951586 (+ 500us) client_negotiation.cc:769] Sending connection context
0807 11:29:30.951941 (+ 355us) client_negotiation.cc:240] Negotiation successful
0807 11:29:30.952441 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":400,"threads_started":1}
I20260807 11:29:30.955132 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:30.928218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:45585 (local address 127.24.103.126:39055)
0807 11:29:30.928797 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:30.928803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:30.929949 (+ 1146us) server_negotiation.cc:407] Connection header received
0807 11:29:30.932372 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:30.932381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:30.932522 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:30.932689 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:30.935231 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.936373 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.939282 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.939717 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.939868 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:30.945239 (+ 5371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:30.945249 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:30.945256 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:30.945313 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:30.948704 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:30.949549 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:30.949566 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:30.949572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:30.949754 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:30.950402 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:30.950408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:30.950411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:30.950802 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:30.950959 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:30.954298 (+ 3339us) server_negotiation.cc:299] Negotiation successful
0807 11:29:30.954628 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":277,"threads_started":1}
I20260807 11:29:30.987317 27473 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:34597
I20260807 11:29:30.987336 27625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:34597 every 8 connection(s)
I20260807 11:29:30.989078 27473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:29:30.989646 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:30.962430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:39839 (local address 127.24.103.126:39055)
0807 11:29:30.962729 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:30.962737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:30.973324 (+ 10587us) server_negotiation.cc:407] Connection header received
0807 11:29:30.973628 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:30.973636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:30.973766 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:30.973889 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:30.975462 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.976716 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.978810 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.979488 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.979650 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:30.982810 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:30.982815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:30.982819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:30.982855 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:30.986410 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:30.987516 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:30.987521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:30.987526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:30.987704 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:30.988165 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:30.988169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:30.988171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:30.988464 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:30.988611 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:30.989091 (+ 480us) server_negotiation.cc:299] Negotiation successful
0807 11:29:30.989363 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260807 11:29:30.989948 27591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:30.962429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39055 (local address 127.24.103.67:39839)
0807 11:29:30.973133 (+ 10704us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:30.973158 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:30.973464 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:30.974058 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:30.974064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:30.974718 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:30.975229 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:30.975249 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.976888 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.976891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:30.978645 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:30.978659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:30.979721 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:30.979731 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:30.979924 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:30.980871 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:30.980907 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:30.982549 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:30.986581 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:30.986593 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:30.986600 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:30.987367 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:30.987819 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:30.987824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:30.987827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:30.987977 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:30.988598 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:30.988603 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:30.988824 (+ 221us) client_negotiation.cc:769] Sending connection context
0807 11:29:30.988981 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 11:29:30.989360 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10524,"thread_start_us":10324,"threads_started":1}
I20260807 11:29:30.991658 27626 heartbeater.cc:347] Connected to a master server at 127.24.103.126:39055
I20260807 11:29:30.993285 27626 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:30.994019 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 27473
I20260807 11:29:30.994189 27626 heartbeater.cc:511] Master 127.24.103.126:39055 requested a full tablet report, sending...
I20260807 11:29:30.994242 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:29:30.994849 24989 external_mini_cluster.cc:1428] Setting key 6f1e3a15f47bc7374f9b1ef6fa3b5cd3
I20260807 11:29:30.996022 27057 ts_manager.cc:194] Registered new tserver with Master: 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597)
I20260807 11:29:30.997294 27057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:45585
I20260807 11:29:30.998550 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:29:31 dist-test-slave-wl2h krb5kdc[27005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 11:29:31 dist-test-slave-wl2h krb5kdc[27005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:31.057384 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.041554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.24.103.126:39055)
0807 11:29:31.041815 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.041821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.042186 (+ 365us) server_negotiation.cc:407] Connection header received
0807 11:29:31.042358 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.042362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.042433 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.042523 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:31.044147 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.044935 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.046214 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.046635 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.046758 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.051701 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:31.051706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:31.051710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:31.051749 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:31.054706 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:31.055475 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:31.055479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:31.055481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:31.055614 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:31.056021 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:31.056024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:31.056026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:31.056293 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:31.056501 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.056833 (+ 332us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.056995 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 11:29:31.061723 27059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:29:31.068686 27059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:29:31.094873 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.082057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37821 (local address 127.0.0.1:56674)
0807 11:29:31.082974 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.083061 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.083275 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.086858 (+ 3583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.086865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.086893 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:31.087508 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.087521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.089516 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.089523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.092390 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.092403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.093730 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.093736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.093951 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.094020 (+ 69us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.094189 (+ 169us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.094390 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":160,"thread_start_us":403,"threads_started":1}
I20260807 11:29:31.095103 27645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.082837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.24.103.66:36761)
0807 11:29:31.085252 (+ 2415us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.085265 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.085305 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:29:31.085440 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.085451 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.086504 (+ 1053us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.086797 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:31.087716 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.089248 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.092512 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.093893 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.094101 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.094300 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.094588 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.094752 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2237,"thread_start_us":2103,"threads_started":1}
I20260807 11:29:31.095032 27646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.082488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:36761 (local address 127.0.0.1:45180)
0807 11:29:31.084108 (+ 1620us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.084129 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.084280 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.086790 (+ 2510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.086798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.086835 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:31.087507 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.087521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.089423 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.089431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.092311 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.092337 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.094198 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.094204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.094381 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.094410 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.094491 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.094637 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1412,"mutex_wait_us":297,"thread_start_us":2838,"threads_started":1}
I20260807 11:29:31.096340 27643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.082221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.24.103.65:37821)
0807 11:29:31.086236 (+ 4015us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.086246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.086271 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:29:31.086364 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.086371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.086616 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.087733 (+ 1117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:31.087976 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.089376 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.092559 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.093518 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.093698 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.093937 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.094444 (+ 507us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.096055 (+ 1611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3839,"thread_start_us":3665,"threads_started":1}
I20260807 11:29:31.096586 27647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.084071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.24.103.67:34597)
0807 11:29:31.085115 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.085129 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.085739 (+ 610us) server_negotiation.cc:407] Connection header received
0807 11:29:31.085848 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.085857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.086107 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.086358 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:31.087926 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.089824 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.092586 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.093983 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.094107 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.094343 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.096013 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.096239 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":499,"threads_started":1}
I20260807 11:29:31.097079 27644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.083376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:34597 (local address 127.0.0.1:52950)
0807 11:29:31.085577 (+ 2201us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.085594 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.085722 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.086446 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.086454 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.086484 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:31.087648 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.087669 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.089987 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.089993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.092356 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.092372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.095200 (+ 2828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.095209 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.095378 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.095413 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.095511 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.095671 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2084,"mutex_wait_us":76,"thread_start_us":960,"threads_started":1}
I20260807 11:29:31.100064 27389 tablet_service.cc:1467] Processing CreateTablet for tablet 71881c3b9ae440a6820bb767d22499d2 (DEFAULT_TABLE table=test-table [id=6e3e43aa231c4dcf9a5c17651e8f63fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:31.100064 27223 tablet_service.cc:1467] Processing CreateTablet for tablet 71881c3b9ae440a6820bb767d22499d2 (DEFAULT_TABLE table=test-table [id=6e3e43aa231c4dcf9a5c17651e8f63fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:31.100359 27549 tablet_service.cc:1467] Processing CreateTablet for tablet 71881c3b9ae440a6820bb767d22499d2 (DEFAULT_TABLE table=test-table [id=6e3e43aa231c4dcf9a5c17651e8f63fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:31.101652 27223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71881c3b9ae440a6820bb767d22499d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:31.101814 27389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71881c3b9ae440a6820bb767d22499d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:31.102125 27549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71881c3b9ae440a6820bb767d22499d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:31.114550 27649 tablet_bootstrap.cc:492] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: Bootstrap starting.
I20260807 11:29:31.116772 27651 tablet_bootstrap.cc:492] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: Bootstrap starting.
I20260807 11:29:31.118456 27653 tablet_bootstrap.cc:492] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: Bootstrap starting.
I20260807 11:29:31.118711 27649 tablet_bootstrap.cc:654] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:31.120154 27649 log.cc:824] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:31.120421 27651 tablet_bootstrap.cc:654] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:31.121754 27653 tablet_bootstrap.cc:654] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:31.122207 27651 log.cc:824] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:31.122941 27653 log.cc:824] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:31.124198 27649 tablet_bootstrap.cc:492] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: No bootstrap required, opened a new log
I20260807 11:29:31.124502 27649 ts_tablet_manager.cc:1397] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 11:29:31.125363 27651 tablet_bootstrap.cc:492] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: No bootstrap required, opened a new log
I20260807 11:29:31.125630 27651 ts_tablet_manager.cc:1397] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:29:31.129434 27653 tablet_bootstrap.cc:492] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: No bootstrap required, opened a new log
I20260807 11:29:31.129799 27653 ts_tablet_manager.cc:1397] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260807 11:29:31.135519 27649 raft_consensus.cc:348] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } }
I20260807 11:29:31.136006 27649 raft_consensus.cc:374] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:31.136103 27649 raft_consensus.cc:729] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48be9ef638fa43e390b78079c00d0159, State: Initialized, Role: FOLLOWER
I20260807 11:29:31.136778 27649 consensus_queue.cc:260] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } }
I20260807 11:29:31.136857 27651 raft_consensus.cc:348] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } }
I20260807 11:29:31.137257 27651 raft_consensus.cc:374] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:31.137384 27651 raft_consensus.cc:729] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4837e34e7d4c4af9b0df7a291c67e581, State: Initialized, Role: FOLLOWER
I20260807 11:29:31.138273 27626 heartbeater.cc:503] Master 127.24.103.126:39055 was elected leader, sending a full tablet report...
I20260807 11:29:31.138145 27651 consensus_queue.cc:260] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } }
I20260807 11:29:31.138408 27653 raft_consensus.cc:348] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } }
I20260807 11:29:31.138914 27653 raft_consensus.cc:374] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:31.139034 27653 raft_consensus.cc:729] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48f3035cbf447eb9e1b6a9f97dc3ae4, State: Initialized, Role: FOLLOWER
I20260807 11:29:31.139312 27649 ts_tablet_manager.cc:1428] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260807 11:29:31.139760 27653 consensus_queue.cc:260] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } }
I20260807 11:29:31.141738 27469 heartbeater.cc:503] Master 127.24.103.126:39055 was elected leader, sending a full tablet report...
I20260807 11:29:31.142175 27653 ts_tablet_manager.cc:1428] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260807 11:29:31.143301 27659 raft_consensus.cc:482] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:29:31.143707 27651 ts_tablet_manager.cc:1428] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260807 11:29:31.143599 27659 raft_consensus.cc:504] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } }
W20260807 11:29:31.146572 27303 tablet.cc:2369] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:29:31.147001 27659 leader_election.cc:288] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821), 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597)
I20260807 11:29:31.154769 27643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.147555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:58235 (local address 127.24.103.65:37821)
0807 11:29:31.147757 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.147763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.148025 (+ 262us) server_negotiation.cc:407] Connection header received
0807 11:29:31.148488 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.148495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.148587 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.148657 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:31.149610 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.150440 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.152646 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.153694 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.153896 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.153974 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.154327 (+ 353us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.154470 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 11:29:31.154860 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.145914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37821 (local address 127.24.103.66:58235)
0807 11:29:31.147916 (+ 2002us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.147943 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.148141 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.148886 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.148898 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.148957 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:31.149427 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.149439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.150586 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.150592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.152483 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.152496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.153920 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.153927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.154155 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.154194 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.154292 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.154434 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1859,"mutex_wait_us":1,"thread_start_us":1687,"threads_started":1}
I20260807 11:29:31.154945 27647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.148481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:34095 (local address 127.24.103.67:34597)
0807 11:29:31.148812 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.148819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.148840 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:29:31.149007 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.149015 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.149105 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.149212 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:31.149931 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.150835 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.152805 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.153961 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.154150 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.154224 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.154553 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.154689 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260807 11:29:31.155521 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.148113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:34597 (local address 127.24.103.66:34095)
0807 11:29:31.148758 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.148772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.148864 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.149258 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.149261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.149282 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:31.149743 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.149756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.150985 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.150989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.152577 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.152596 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.154181 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.154187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.154375 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.154405 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.155090 (+ 685us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.155233 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":359,"threads_started":1}
I20260807 11:29:31.156155 27245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71881c3b9ae440a6820bb767d22499d2" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4837e34e7d4c4af9b0df7a291c67e581" is_pre_election: true
I20260807 11:29:31.156651 27245 raft_consensus.cc:2454] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 0.
I20260807 11:29:31.156564 27570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71881c3b9ae440a6820bb767d22499d2" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48be9ef638fa43e390b78079c00d0159" is_pre_election: true
I20260807 11:29:31.157157 27570 raft_consensus.cc:2454] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 0.
I20260807 11:29:31.157374 27327 leader_election.cc:302] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4837e34e7d4c4af9b0df7a291c67e581, f48f3035cbf447eb9e1b6a9f97dc3ae4; no voters:
I20260807 11:29:31.157883 27659 raft_consensus.cc:2781] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:29:31.158102 27659 raft_consensus.cc:482] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:29:31.158293 27659 raft_consensus.cc:3037] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:31.160941 27659 raft_consensus.cc:504] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } }
I20260807 11:29:31.161618 27659 leader_election.cc:288] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 election: Requested vote from peers 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821), 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597)
I20260807 11:29:31.162120 27245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71881c3b9ae440a6820bb767d22499d2" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4837e34e7d4c4af9b0df7a291c67e581"
I20260807 11:29:31.162290 27570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71881c3b9ae440a6820bb767d22499d2" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48be9ef638fa43e390b78079c00d0159"
I20260807 11:29:31.162387 27245 raft_consensus.cc:3037] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:31.162531 27570 raft_consensus.cc:3037] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:31.165207 27570 raft_consensus.cc:2454] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 1.
I20260807 11:29:31.165201 27245 raft_consensus.cc:2454] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 1.
I20260807 11:29:31.165673 27327 leader_election.cc:302] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48be9ef638fa43e390b78079c00d0159, f48f3035cbf447eb9e1b6a9f97dc3ae4; no voters:
I20260807 11:29:31.166035 27659 raft_consensus.cc:2781] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:31.166540 27659 raft_consensus.cc:686] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 LEADER]: Becoming Leader. State: Replica: f48f3035cbf447eb9e1b6a9f97dc3ae4, State: Running, Role: LEADER
I20260807 11:29:31.167193 27659 consensus_queue.cc:237] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } }
I20260807 11:29:31.171972 27056 catalog_manager.cc:5495] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66). New cstate: current_term: 1 leader_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } health_report { overall_health: HEALTHY } } }
W20260807 11:29:31.183528 27470 tablet.cc:2369] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 11:29:31 dist-test-slave-wl2h krb5kdc[27005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102169, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:31.237574 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.221857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39055 (local address 127.0.0.1:46948)
0807 11:29:31.222533 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.222550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.222850 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.223269 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.223273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.224040 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:31.224493 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.224507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.225651 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.225657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.227340 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.227354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.228184 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.228191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.228370 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.229109 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:31.229137 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:31.231876 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:31.234982 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:31.234993 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:31.234999 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:31.235542 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:31.235881 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:31.235885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:31.235887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:31.236023 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:31.236533 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:31.236539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:31.236746 (+ 207us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.236890 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.237097 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":386,"threads_started":1}
I20260807 11:29:31.237622 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.221828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.24.103.126:39055)
0807 11:29:31.222022 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.222029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.222759 (+ 730us) server_negotiation.cc:407] Connection header received
0807 11:29:31.222981 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.222992 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.223118 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.223232 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:31.224690 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.225503 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.227495 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.227968 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.228076 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.232130 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:31.232135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:31.232138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:31.232174 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:31.234839 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:31.235654 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:31.235658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:31.235661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:31.235782 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:31.236137 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:31.236140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:31.236142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:31.236405 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:31.236512 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.237051 (+ 539us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.237361 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260807 11:29:31.241962 27627 tablet.cc:2369] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:29:31.243229 27056 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:46948:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 11:29:31.244339 27056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:29:31.251791 27223 tablet_service.cc:1467] Processing CreateTablet for tablet 59ce180126e04d96a29059f054a0ea5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=618f4d9db63e4bd8b30c417ad20633ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:29:31.251967 27389 tablet_service.cc:1467] Processing CreateTablet for tablet 59ce180126e04d96a29059f054a0ea5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=618f4d9db63e4bd8b30c417ad20633ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:29:31.252086 27549 tablet_service.cc:1467] Processing CreateTablet for tablet 59ce180126e04d96a29059f054a0ea5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=618f4d9db63e4bd8b30c417ad20633ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:29:31.252806 27223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ce180126e04d96a29059f054a0ea5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:31.252934 27389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ce180126e04d96a29059f054a0ea5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:31.252934 27549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ce180126e04d96a29059f054a0ea5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:31.258435 27651 tablet_bootstrap.cc:492] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: Bootstrap starting.
I20260807 11:29:31.260115 27651 tablet_bootstrap.cc:654] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:31.260244 27653 tablet_bootstrap.cc:492] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4: Bootstrap starting.
I20260807 11:29:31.262138 27653 tablet_bootstrap.cc:654] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:31.262708 27651 tablet_bootstrap.cc:492] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: No bootstrap required, opened a new log
I20260807 11:29:31.262910 27651 ts_tablet_manager.cc:1397] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260807 11:29:31.263177 27649 tablet_bootstrap.cc:492] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159: Bootstrap starting.
I20260807 11:29:31.263717 27651 raft_consensus.cc:348] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.263926 27651 raft_consensus.cc:374] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:31.264068 27651 raft_consensus.cc:729] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4837e34e7d4c4af9b0df7a291c67e581, State: Initialized, Role: FOLLOWER
I20260807 11:29:31.264328 27651 consensus_queue.cc:260] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.264901 27649 tablet_bootstrap.cc:654] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:31.265017 27651 ts_tablet_manager.cc:1428] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260807 11:29:31.264806 27658 tablet_replica.cc:439] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } }
I20260807 11:29:31.265230 27658 tablet_replica.cc:442] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:31.267421 27649 tablet_bootstrap.cc:492] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159: No bootstrap required, opened a new log
I20260807 11:29:31.267596 27649 ts_tablet_manager.cc:1397] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 11:29:31.268266 27649 raft_consensus.cc:348] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.268491 27649 raft_consensus.cc:374] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:31.268584 27649 raft_consensus.cc:729] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48be9ef638fa43e390b78079c00d0159, State: Initialized, Role: FOLLOWER
I20260807 11:29:31.268829 27649 consensus_queue.cc:260] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.269424 27649 ts_tablet_manager.cc:1428] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 11:29:31.269333 27657 tablet_replica.cc:439] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } }
I20260807 11:29:31.269750 27657 tablet_replica.cc:442] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:31.270567 27653 tablet_bootstrap.cc:492] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4: No bootstrap required, opened a new log
I20260807 11:29:31.270730 27653 ts_tablet_manager.cc:1397] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20260807 11:29:31.271591 27653 raft_consensus.cc:348] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.271834 27653 raft_consensus.cc:374] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:31.271941 27653 raft_consensus.cc:729] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48f3035cbf447eb9e1b6a9f97dc3ae4, State: Initialized, Role: FOLLOWER
I20260807 11:29:31.272217 27653 consensus_queue.cc:260] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.272710 27659 tablet_replica.cc:439] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } }
I20260807 11:29:31.273031 27659 tablet_replica.cc:442] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:31.272887 27653 ts_tablet_manager.cc:1428] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260807 11:29:31.323719 27658 raft_consensus.cc:482] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:29:31.323961 27658 raft_consensus.cc:504] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.325881 27658 leader_election.cc:288] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66:36761), 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597)
I20260807 11:29:31.332739 27645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.326341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:43201 (local address 127.24.103.66:36761)
0807 11:29:31.326555 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.326561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.327429 (+ 868us) server_negotiation.cc:407] Connection header received
0807 11:29:31.327676 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.327684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.327758 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.327860 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:31.328479 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.329200 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.330937 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.331786 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.331996 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.332089 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.332347 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.332473 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:29:31.332885 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.326300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:36761 (local address 127.24.103.65:43201)
0807 11:29:31.327258 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.327274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.327515 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.327882 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.327886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.327914 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:31.328313 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.328323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.329337 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.329343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.330774 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.330786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.332005 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.332024 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.332187 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.332211 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.332321 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.332452 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"spinlock_wait_cycles":60672,"thread_start_us":554,"threads_started":1}
I20260807 11:29:31.334039 27647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.326345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:59083 (local address 127.24.103.67:34597)
0807 11:29:31.326556 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.326564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.328064 (+ 1500us) server_negotiation.cc:407] Connection header received
0807 11:29:31.328303 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.328307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.328473 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.328615 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:31.329191 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.330033 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.331983 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.332948 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.333107 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.333191 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.333744 (+ 553us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.333845 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:29:31.333930 27416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ce180126e04d96a29059f054a0ea5d" candidate_uuid: "4837e34e7d4c4af9b0df7a291c67e581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" is_pre_election: true
I20260807 11:29:31.334352 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.327070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:34597 (local address 127.24.103.65:59083)
0807 11:29:31.327940 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.327954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.328136 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.328629 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.328635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.328662 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:31.329006 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.329016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.330184 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.330190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.331807 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.331820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.333333 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.333338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.333562 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.333597 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.333722 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.333899 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":507,"threads_started":1}
I20260807 11:29:31.334472 27416 raft_consensus.cc:2454] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4837e34e7d4c4af9b0df7a291c67e581 in term 0.
I20260807 11:29:31.335119 27570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ce180126e04d96a29059f054a0ea5d" candidate_uuid: "4837e34e7d4c4af9b0df7a291c67e581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48be9ef638fa43e390b78079c00d0159" is_pre_election: true
I20260807 11:29:31.335160 27165 leader_election.cc:302] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4837e34e7d4c4af9b0df7a291c67e581, f48f3035cbf447eb9e1b6a9f97dc3ae4; no voters:
I20260807 11:29:31.335376 27570 raft_consensus.cc:2454] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4837e34e7d4c4af9b0df7a291c67e581 in term 0.
I20260807 11:29:31.335590 27658 raft_consensus.cc:2781] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:29:31.335757 27658 raft_consensus.cc:482] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:29:31.335906 27658 raft_consensus.cc:3037] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:31.338734 27658 raft_consensus.cc:504] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.339548 27658 leader_election.cc:288] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [CANDIDATE]: Term 1 election: Requested vote from peers f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66:36761), 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597)
I20260807 11:29:31.340031 27416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ce180126e04d96a29059f054a0ea5d" candidate_uuid: "4837e34e7d4c4af9b0df7a291c67e581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4"
I20260807 11:29:31.340075 27570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ce180126e04d96a29059f054a0ea5d" candidate_uuid: "4837e34e7d4c4af9b0df7a291c67e581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48be9ef638fa43e390b78079c00d0159"
I20260807 11:29:31.340269 27416 raft_consensus.cc:3037] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:31.340325 27570 raft_consensus.cc:3037] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:31.342904 27570 raft_consensus.cc:2454] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4837e34e7d4c4af9b0df7a291c67e581 in term 1.
I20260807 11:29:31.342949 27416 raft_consensus.cc:2454] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4837e34e7d4c4af9b0df7a291c67e581 in term 1.
I20260807 11:29:31.343387 27163 leader_election.cc:302] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4837e34e7d4c4af9b0df7a291c67e581, 48be9ef638fa43e390b78079c00d0159; no voters:
I20260807 11:29:31.343701 27658 raft_consensus.cc:2781] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:31.344148 27658 raft_consensus.cc:686] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 LEADER]: Becoming Leader. State: Replica: 4837e34e7d4c4af9b0df7a291c67e581, State: Running, Role: LEADER
I20260807 11:29:31.344697 27658 consensus_queue.cc:237] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.348367 27680 tablet_replica.cc:439] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: TxnStatusTablet state changed. Reason: New leader 4837e34e7d4c4af9b0df7a291c67e581. Latest consensus state: current_term: 1 leader_uuid: "4837e34e7d4c4af9b0df7a291c67e581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } }
I20260807 11:29:31.348821 27680 tablet_replica.cc:442] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:29:31.349502 27682 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 11:29:31.349758 27682 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 11:29:31.349673 27056 catalog_manager.cc:5495] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "4837e34e7d4c4af9b0df7a291c67e581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } health_report { overall_health: UNKNOWN } } }
I20260807 11:29:31.378142 27643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.370928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.24.103.65:37821)
0807 11:29:31.371200 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.371206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.371353 (+ 147us) server_negotiation.cc:407] Connection header received
0807 11:29:31.371665 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.371669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.371761 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.371923 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:31.372322 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.373246 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.375248 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.375677 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.375798 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.376446 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:31.377248 (+ 802us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:31.377415 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.377607 (+ 192us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.377729 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260807 11:29:31.378368 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.370925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37821 (local address 127.0.0.1:56676)
0807 11:29:31.371245 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.371267 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.371448 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.371942 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.371948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.371960 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:29:31.372177 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.372186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.373406 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.373412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.375059 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.375076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.375856 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.375865 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.376098 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.376174 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:29:31.377388 (+ 1214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:29:31.377418 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.377579 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.377906 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 11:29:31.388186 27416 raft_consensus.cc:1261] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Refusing update from remote peer 4837e34e7d4c4af9b0df7a291c67e581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:31.388203 27570 raft_consensus.cc:1261] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Refusing update from remote peer 4837e34e7d4c4af9b0df7a291c67e581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:31.389117 27658 consensus_queue.cc:1035] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:31.389539 27680 consensus_queue.cc:1035] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:31.397842 27657 tablet_replica.cc:439] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159: TxnStatusTablet state changed. Reason: New leader 4837e34e7d4c4af9b0df7a291c67e581. Latest consensus state: current_term: 1 leader_uuid: "4837e34e7d4c4af9b0df7a291c67e581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } }
I20260807 11:29:31.398167 27657 tablet_replica.cc:442] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:31.404722 27659 tablet_replica.cc:439] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4: TxnStatusTablet state changed. Reason: New leader 4837e34e7d4c4af9b0df7a291c67e581. Latest consensus state: current_term: 1 leader_uuid: "4837e34e7d4c4af9b0df7a291c67e581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } }
I20260807 11:29:31.405249 27659 tablet_replica.cc:442] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:31.409341 27658 tablet_replica.cc:439] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4837e34e7d4c4af9b0df7a291c67e581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } }
I20260807 11:29:31.409679 27658 tablet_replica.cc:442] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:29:31.410259 27658 tablet_replica.cc:439] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4837e34e7d4c4af9b0df7a291c67e581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } }
I20260807 11:29:31.410526 27658 tablet_replica.cc:442] T 59ce180126e04d96a29059f054a0ea5d P 4837e34e7d4c4af9b0df7a291c67e581: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:29:31.411711 27657 tablet_replica.cc:439] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4837e34e7d4c4af9b0df7a291c67e581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } }
I20260807 11:29:31.411980 27657 tablet_replica.cc:442] T 59ce180126e04d96a29059f054a0ea5d P 48be9ef638fa43e390b78079c00d0159: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:31.417865 27659 tablet_replica.cc:439] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4837e34e7d4c4af9b0df7a291c67e581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } } }
I20260807 11:29:31.418219 27659 tablet_replica.cc:442] T 59ce180126e04d96a29059f054a0ea5d P f48f3035cbf447eb9e1b6a9f97dc3ae4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:29:31.428573 27645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.422478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.24.103.66:36761)
0807 11:29:31.422661 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.422667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.422870 (+ 203us) server_negotiation.cc:407] Connection header received
0807 11:29:31.423193 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.423197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.423275 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.423438 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:31.423824 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.424624 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.426378 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.426783 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.426935 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.427420 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:31.427949 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:31.428128 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.428183 (+ 55us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.428268 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 11:29:31.446807 27643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.441098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:40169 (local address 127.24.103.65:37821)
0807 11:29:31.441366 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.441372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.441630 (+ 258us) server_negotiation.cc:407] Connection header received
0807 11:29:31.441875 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.441887 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.441953 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.442054 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:31.442618 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.443470 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.444913 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.445355 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.445489 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.446039 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:31.446149 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:31.446249 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.446392 (+ 143us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.446472 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 11:29:31.446904 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.440700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37821 (local address 127.24.103.66:40169)
0807 11:29:31.441513 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.441531 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.441715 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.442079 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.442084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.442096 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:29:31.442451 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.442460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.443616 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.443622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.444767 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.444778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.445555 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.445564 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.445696 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.445748 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:29:31.446252 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:29:31.446267 (+ 15us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.446370 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.446512 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":465,"threads_started":1}
I20260807 11:29:31.458251 27570 raft_consensus.cc:1261] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Refusing update from remote peer f48f3035cbf447eb9e1b6a9f97dc3ae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:31.458774 27245 raft_consensus.cc:1261] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Refusing update from remote peer f48f3035cbf447eb9e1b6a9f97dc3ae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:31.459046 27663 consensus_queue.cc:1035] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:31.459578 27659 consensus_queue.cc:1035] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:31.473876 27690 mvcc.cc:205] Tried to move back new op lower bound from 7315874494283988992 to 7315874493105098752. Current Snapshot: MvccSnapshot[applied={T|T < 7315874494283988992}]
I20260807 11:29:31.475983 27689 mvcc.cc:205] Tried to move back new op lower bound from 7315874494283988992 to 7315874493105098752. Current Snapshot: MvccSnapshot[applied={T|T < 7315874494283988992}]
I20260807 11:29:31.511401 27645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.503714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:43077 (local address 127.24.103.66:36761)
0807 11:29:31.503913 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.503921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.504602 (+ 681us) server_negotiation.cc:407] Connection header received
0807 11:29:31.504869 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.504875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.504952 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.505061 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:31.505742 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.506752 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.508911 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.509379 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.509535 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.510118 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:31.510232 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:31.510346 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.510922 (+ 576us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.511050 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:29:31.511646 27705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.503578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:36761 (local address 127.24.103.65:43077)
0807 11:29:31.504487 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.504504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.504692 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.505087 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.505092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.505106 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:29:31.505530 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.505544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.506948 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.506953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.508753 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.508767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.509577 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.509585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.509801 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.509891 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:29:31.510730 (+ 839us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:29:31.510744 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.510873 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.511093 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":523,"threads_started":1}
I20260807 11:29:31.557124 27057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:31.557448 27057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:31.561934 27057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b: Sending DeleteTablet for 3 replicas of tablet 71881c3b9ae440a6820bb767d22499d2
I20260807 11:29:31.563498 27389 tablet_service.cc:1514] Processing DeleteTablet for tablet 71881c3b9ae440a6820bb767d22499d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:31 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:45180
I20260807 11:29:31.563611 27549 tablet_service.cc:1514] Processing DeleteTablet for tablet 71881c3b9ae440a6820bb767d22499d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:31 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:52950
I20260807 11:29:31.564548 27223 tablet_service.cc:1514] Processing DeleteTablet for tablet 71881c3b9ae440a6820bb767d22499d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:31 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:56674
I20260807 11:29:31.570744 27057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:29:31.572306 27057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:29:31.574527 27709 tablet_replica.cc:330] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: stopping tablet replica
I20260807 11:29:31.575003 27709 raft_consensus.cc:2229] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:29:31.575789 27709 raft_consensus.cc:2258] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:31.583814 27389 tablet_service.cc:1467] Processing CreateTablet for tablet 9211c8e6bb234745ae72b9568826f90c (DEFAULT_TABLE table=test-table [id=70c73332fdf34f57a079213ae6f267ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:31.584299 27389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9211c8e6bb234745ae72b9568826f90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:31.584214 27548 tablet_service.cc:1467] Processing CreateTablet for tablet 9211c8e6bb234745ae72b9568826f90c (DEFAULT_TABLE table=test-table [id=70c73332fdf34f57a079213ae6f267ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:31.584666 27548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9211c8e6bb234745ae72b9568826f90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:31.587584 27221 tablet_service.cc:1467] Processing CreateTablet for tablet 9211c8e6bb234745ae72b9568826f90c (DEFAULT_TABLE table=test-table [id=70c73332fdf34f57a079213ae6f267ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:31.588269 27221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9211c8e6bb234745ae72b9568826f90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:31.591715 27709 ts_tablet_manager.cc:1905] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:31.596134 27649 tablet_bootstrap.cc:492] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159: Bootstrap starting.
I20260807 11:29:31.597913 27649 tablet_bootstrap.cc:654] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:31.598284 27653 tablet_bootstrap.cc:492] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Bootstrap starting.
I20260807 11:29:31.599097 27651 tablet_bootstrap.cc:492] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581: Bootstrap starting.
I20260807 11:29:31.600091 27653 tablet_bootstrap.cc:654] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:31.601198 27651 tablet_bootstrap.cc:654] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:31.608062 27649 tablet_bootstrap.cc:492] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159: No bootstrap required, opened a new log
I20260807 11:29:31.608322 27649 ts_tablet_manager.cc:1397] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260807 11:29:31.609303 27649 raft_consensus.cc:348] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.609710 27649 raft_consensus.cc:374] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:31.609840 27649 raft_consensus.cc:729] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48be9ef638fa43e390b78079c00d0159, State: Initialized, Role: FOLLOWER
I20260807 11:29:31.610234 27649 consensus_queue.cc:260] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.614127 27649 ts_tablet_manager.cc:1428] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260807 11:29:31.614276 27709 ts_tablet_manager.cc:1918] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:29:31.614470 27709 log.cc:1196] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/71881c3b9ae440a6820bb767d22499d2
I20260807 11:29:31.619153 27709 ts_tablet_manager.cc:1939] T 71881c3b9ae440a6820bb767d22499d2 P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting consensus metadata
I20260807 11:29:31.620010 27711 tablet_replica.cc:330] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: stopping tablet replica
I20260807 11:29:31.620332 27651 tablet_bootstrap.cc:492] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581: No bootstrap required, opened a new log
I20260807 11:29:31.620553 27651 ts_tablet_manager.cc:1397] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.009s
I20260807 11:29:31.620743 27712 tablet_replica.cc:330] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: stopping tablet replica
I20260807 11:29:31.621268 27711 raft_consensus.cc:2229] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:31.621248 27043 catalog_manager.cc:4841] TS f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66:36761): tablet 71881c3b9ae440a6820bb767d22499d2 (table test-table [id=6e3e43aa231c4dcf9a5c17651e8f63fe]) successfully deleted
I20260807 11:29:31.621376 27651 raft_consensus.cc:348] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.621652 27651 raft_consensus.cc:374] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:31.621744 27651 raft_consensus.cc:729] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4837e34e7d4c4af9b0df7a291c67e581, State: Initialized, Role: FOLLOWER
I20260807 11:29:31.621966 27712 raft_consensus.cc:2229] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:31.622133 27651 consensus_queue.cc:260] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.622681 27712 raft_consensus.cc:2258] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:31.623528 27711 raft_consensus.cc:2258] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:31.624186 27651 ts_tablet_manager.cc:1428] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260807 11:29:31.625406 27711 ts_tablet_manager.cc:1905] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:31.626113 27712 ts_tablet_manager.cc:1905] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:31.632261 27653 tablet_bootstrap.cc:492] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4: No bootstrap required, opened a new log
I20260807 11:29:31.632385 27712 ts_tablet_manager.cc:1918] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:29:31.632458 27653 ts_tablet_manager.cc:1397] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.003s
I20260807 11:29:31.632573 27712 log.cc:1196] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/71881c3b9ae440a6820bb767d22499d2
I20260807 11:29:31.632762 27711 ts_tablet_manager.cc:1918] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:29:31.632926 27711 log.cc:1196] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/71881c3b9ae440a6820bb767d22499d2
I20260807 11:29:31.633157 27712 ts_tablet_manager.cc:1939] T 71881c3b9ae440a6820bb767d22499d2 P 4837e34e7d4c4af9b0df7a291c67e581: Deleting consensus metadata
I20260807 11:29:31.633450 27711 ts_tablet_manager.cc:1939] T 71881c3b9ae440a6820bb767d22499d2 P 48be9ef638fa43e390b78079c00d0159: Deleting consensus metadata
I20260807 11:29:31.633401 27653 raft_consensus.cc:348] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.633708 27653 raft_consensus.cc:374] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:31.633790 27653 raft_consensus.cc:729] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48f3035cbf447eb9e1b6a9f97dc3ae4, State: Initialized, Role: FOLLOWER
I20260807 11:29:31.634042 27653 consensus_queue.cc:260] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.635108 27041 catalog_manager.cc:4841] TS 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597): tablet 71881c3b9ae440a6820bb767d22499d2 (table test-table [id=6e3e43aa231c4dcf9a5c17651e8f63fe]) successfully deleted
I20260807 11:29:31.635393 27041 catalog_manager.cc:4841] TS 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821): tablet 71881c3b9ae440a6820bb767d22499d2 (table test-table [id=6e3e43aa231c4dcf9a5c17651e8f63fe]) successfully deleted
I20260807 11:29:31.634653 27653 ts_tablet_manager.cc:1428] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260807 11:29:31.862123 27657 raft_consensus.cc:482] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:29:31.862398 27657 raft_consensus.cc:504] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.864269 27657 leader_election.cc:288] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821), f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66:36761)
I20260807 11:29:31.880771 27645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.866117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:54677 (local address 127.24.103.66:36761)
0807 11:29:31.866325 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.866331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.873269 (+ 6938us) server_negotiation.cc:407] Connection header received
0807 11:29:31.873552 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.873560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.873634 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.873695 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:31.874474 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.875616 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.878161 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.879595 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.879800 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.879889 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.880281 (+ 392us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.880451 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:29:31.880828 27716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.865231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:36761 (local address 127.24.103.67:54677)
0807 11:29:31.873136 (+ 7905us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.873157 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.873383 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.873898 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.873902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.873927 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:31.874277 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.874291 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.875812 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.875817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.877980 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.877997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.879821 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.879828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.880009 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.880107 (+ 98us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.880230 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.880397 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7764,"mutex_wait_us":6,"thread_start_us":7273,"threads_started":1}
I20260807 11:29:31.880944 27643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.865042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:59283 (local address 127.24.103.65:37821)
0807 11:29:31.865243 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.865250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.872959 (+ 7709us) server_negotiation.cc:407] Connection header received
0807 11:29:31.873577 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.873583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.873667 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.873733 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:31.874661 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.875765 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.878433 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.879746 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.879902 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.879978 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.880463 (+ 485us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.880601 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:29:31.881757 27717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.864611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37821 (local address 127.24.103.67:59283)
0807 11:29:31.872786 (+ 8175us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.872815 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.873371 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:31.873943 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:31.873948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:31.873986 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:31.874443 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.874456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.875982 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.875987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.878244 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:31.878259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.880035 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.880045 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:31.880270 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.880304 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 11:29:31.881170 (+ 866us) client_negotiation.cc:240] Negotiation successful
0807 11:29:31.881319 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8028,"mutex_wait_us":68,"spinlock_wait_cycles":4480,"thread_start_us":8452,"threads_started":1}
I20260807 11:29:31.882277 27416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211c8e6bb234745ae72b9568826f90c" candidate_uuid: "48be9ef638fa43e390b78079c00d0159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" is_pre_election: true
I20260807 11:29:31.882465 27244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211c8e6bb234745ae72b9568826f90c" candidate_uuid: "48be9ef638fa43e390b78079c00d0159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4837e34e7d4c4af9b0df7a291c67e581" is_pre_election: true
I20260807 11:29:31.882576 27416 raft_consensus.cc:2454] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48be9ef638fa43e390b78079c00d0159 in term 0.
I20260807 11:29:31.882753 27244 raft_consensus.cc:2454] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48be9ef638fa43e390b78079c00d0159 in term 0.
I20260807 11:29:31.883116 27493 leader_election.cc:302] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48be9ef638fa43e390b78079c00d0159, f48f3035cbf447eb9e1b6a9f97dc3ae4; no voters:
I20260807 11:29:31.883613 27657 raft_consensus.cc:2781] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:29:31.883749 27657 raft_consensus.cc:482] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:29:31.883864 27657 raft_consensus.cc:3037] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:31.887156 27657 raft_consensus.cc:504] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.887907 27657 leader_election.cc:288] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [CANDIDATE]: Term 1 election: Requested vote from peers 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821), f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66:36761)
I20260807 11:29:31.888427 27244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211c8e6bb234745ae72b9568826f90c" candidate_uuid: "48be9ef638fa43e390b78079c00d0159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4837e34e7d4c4af9b0df7a291c67e581"
I20260807 11:29:31.888486 27416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211c8e6bb234745ae72b9568826f90c" candidate_uuid: "48be9ef638fa43e390b78079c00d0159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4"
I20260807 11:29:31.888659 27244 raft_consensus.cc:3037] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:31.888716 27416 raft_consensus.cc:3037] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:31.891788 27416 raft_consensus.cc:2454] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48be9ef638fa43e390b78079c00d0159 in term 1.
I20260807 11:29:31.891858 27244 raft_consensus.cc:2454] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48be9ef638fa43e390b78079c00d0159 in term 1.
I20260807 11:29:31.892314 27493 leader_election.cc:302] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48be9ef638fa43e390b78079c00d0159, f48f3035cbf447eb9e1b6a9f97dc3ae4; no voters:
I20260807 11:29:31.892707 27657 raft_consensus.cc:2781] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:31.895633 27657 raft_consensus.cc:686] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 1 LEADER]: Becoming Leader. State: Replica: 48be9ef638fa43e390b78079c00d0159, State: Running, Role: LEADER
I20260807 11:29:31.896123 27657 consensus_queue.cc:237] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:31.905751 27056 catalog_manager.cc:5495] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48be9ef638fa43e390b78079c00d0159 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "48be9ef638fa43e390b78079c00d0159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } health_report { overall_health: HEALTHY } } }
I20260807 11:29:31.985013 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.978547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.24.103.67:34597)
0807 11:29:31.979334 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.979340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.979361 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:29:31.979466 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:31.979470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:31.979544 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:31.979737 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:31.980253 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.981079 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.982555 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:31.983066 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:31.983213 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:31.983658 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:31.984258 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:31.984426 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:31.984541 (+ 115us) server_negotiation.cc:299] Negotiation successful
0807 11:29:31.984638 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":478,"threads_started":1}
I20260807 11:29:32.006893 27643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.997791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:40265 (local address 127.24.103.65:37821)
0807 11:29:31.998015 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:31.998022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:31.999836 (+ 1814us) server_negotiation.cc:407] Connection header received
0807 11:29:32.000254 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:32.000272 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:32.000504 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:32.000611 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:32.001342 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:32.002422 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:32.004726 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:32.005394 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:32.005515 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:32.006092 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:32.006252 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:32.006338 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:32.006501 (+ 163us) server_negotiation.cc:299] Negotiation successful
0807 11:29:32.006587 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:29:32.007287 27726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:31.997815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37821 (local address 127.24.103.67:40265)
0807 11:29:31.999715 (+ 1900us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:31.999738 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:31.999938 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:32.000664 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:32.000670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:32.000687 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:29:32.001151 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:32.001165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:32.002591 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:32.002597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:32.004555 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:32.004573 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:32.005602 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:32.005610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:32.005815 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:32.005893 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:29:32.006365 (+ 472us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:29:32.006378 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 11:29:32.006468 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 11:29:32.006618 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1682,"thread_start_us":1370,"threads_started":1}
I20260807 11:29:32.017184 27416 raft_consensus.cc:1261] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Refusing update from remote peer 48be9ef638fa43e390b78079c00d0159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:32.017886 27244 raft_consensus.cc:1261] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Refusing update from remote peer 48be9ef638fa43e390b78079c00d0159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:32.018652 27718 consensus_queue.cc:1035] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:32.019671 27657 consensus_queue.cc:1035] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:32.026153 27687 mvcc.cc:205] Tried to move back new op lower bound from 7315874496574619648 to 7315874496091447296. Current Snapshot: MvccSnapshot[applied={T|T < 7315874496574619648}]
I20260807 11:29:32.075103 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:32.056063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:42789 (local address 127.24.103.67:34597)
0807 11:29:32.056264 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:32.056274 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:32.067742 (+ 11468us) server_negotiation.cc:407] Connection header received
0807 11:29:32.068093 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:32.068099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:32.068194 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:32.068263 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:32.069069 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:32.070262 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:32.072672 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:32.073477 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:32.073610 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:32.074226 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:32.074367 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:32.074459 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:32.074655 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 11:29:32.074741 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 11:29:32.075209 27734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:32.056027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:34597 (local address 127.24.103.65:42789)
0807 11:29:32.067580 (+ 11553us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:32.067607 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:32.067890 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:32.068547 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:32.068554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:32.068572 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:29:32.068870 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:32.068883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:32.070477 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:32.070485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:32.072467 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:32.072488 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:32.073681 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:32.073690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:32.073929 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:32.074010 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:29:32.074494 (+ 484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:29:32.074510 (+ 16us) client_negotiation.cc:769] Sending connection context
0807 11:29:32.074609 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 11:29:32.074780 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11339,"thread_start_us":11181,"threads_started":1}
W20260807 11:29:32.076718 27689 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 11:29:33.686820 27056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:33.687191 27056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:33.691869 27056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b: Sending DeleteTablet for 3 replicas of tablet 9211c8e6bb234745ae72b9568826f90c
I20260807 11:29:33.693005 27223 tablet_service.cc:1514] Processing DeleteTablet for tablet 9211c8e6bb234745ae72b9568826f90c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:33 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:56674
I20260807 11:29:33.693120 27389 tablet_service.cc:1514] Processing DeleteTablet for tablet 9211c8e6bb234745ae72b9568826f90c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:33 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:45180
I20260807 11:29:33.693127 27549 tablet_service.cc:1514] Processing DeleteTablet for tablet 9211c8e6bb234745ae72b9568826f90c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:33 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:52950
I20260807 11:29:33.705722 27762 tablet_replica.cc:330] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581: stopping tablet replica
I20260807 11:29:33.705978 27763 tablet_replica.cc:330] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159: stopping tablet replica
I20260807 11:29:33.706324 27763 raft_consensus.cc:2229] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:29:33.706718 27762 raft_consensus.cc:2229] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:33.706998 27762 raft_consensus.cc:2258] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:33.707248 27763 raft_consensus.cc:2258] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:33.708508 27763 ts_tablet_manager.cc:1905] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:33.709154 27762 ts_tablet_manager.cc:1905] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:33.709371 27761 tablet_replica.cc:330] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4: stopping tablet replica
I20260807 11:29:33.709664 27761 raft_consensus.cc:2229] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:33.709956 27761 raft_consensus.cc:2258] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:33.711215 27761 ts_tablet_manager.cc:1905] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:33.715585 27763 ts_tablet_manager.cc:1918] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:29:33.715787 27763 log.cc:1196] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/9211c8e6bb234745ae72b9568826f90c
I20260807 11:29:33.716290 27763 ts_tablet_manager.cc:1939] T 9211c8e6bb234745ae72b9568826f90c P 48be9ef638fa43e390b78079c00d0159: Deleting consensus metadata
I20260807 11:29:33.716357 27762 ts_tablet_manager.cc:1918] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:29:33.716625 27762 log.cc:1196] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/9211c8e6bb234745ae72b9568826f90c
I20260807 11:29:33.717170 27762 ts_tablet_manager.cc:1939] T 9211c8e6bb234745ae72b9568826f90c P 4837e34e7d4c4af9b0df7a291c67e581: Deleting consensus metadata
I20260807 11:29:33.718106 27041 catalog_manager.cc:4841] TS 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597): tablet 9211c8e6bb234745ae72b9568826f90c (table test-table [id=70c73332fdf34f57a079213ae6f267ef]) successfully deleted
I20260807 11:29:33.718925 27041 catalog_manager.cc:4841] TS 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821): tablet 9211c8e6bb234745ae72b9568826f90c (table test-table [id=70c73332fdf34f57a079213ae6f267ef]) successfully deleted
I20260807 11:29:33.719712 27761 ts_tablet_manager.cc:1918] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:29:33.719831 27761 log.cc:1196] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/9211c8e6bb234745ae72b9568826f90c
I20260807 11:29:33.720317 27761 ts_tablet_manager.cc:1939] T 9211c8e6bb234745ae72b9568826f90c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting consensus metadata
I20260807 11:29:33.721359 27043 catalog_manager.cc:4841] TS f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66:36761): tablet 9211c8e6bb234745ae72b9568826f90c (table test-table [id=70c73332fdf34f57a079213ae6f267ef]) successfully deleted
Aug 07 11:29:33 dist-test-slave-wl2h krb5kdc[27005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 11:29:33 dist-test-slave-wl2h krb5kdc[27005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:33.759404 27771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:33.744044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.24.103.126:39055)
0807 11:29:33.744720 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:33.744726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:33.744859 (+ 133us) server_negotiation.cc:407] Connection header received
0807 11:29:33.745117 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:33.745122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:33.745190 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:33.745325 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:33.746740 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:33.747609 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:33.749016 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:33.749470 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:33.749616 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:33.753915 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:33.753921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:33.753924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:33.753960 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:33.756635 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:33.757357 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:33.757364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:33.757368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:33.757527 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:33.757916 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:33.757921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:33.757924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:33.758203 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:33.758464 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:33.758770 (+ 306us) server_negotiation.cc:299] Negotiation successful
0807 11:29:33.759002 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":341,"threads_started":1}
I20260807 11:29:33.763185 27056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:29:33.763867 27056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:29:33.770789 27223 tablet_service.cc:1467] Processing CreateTablet for tablet 1877a476d9594719ab8bcc8c9553283a (DEFAULT_TABLE table=test-table [id=01032cf9058844c1a167e0049f45298c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:33.770991 27389 tablet_service.cc:1467] Processing CreateTablet for tablet 1877a476d9594719ab8bcc8c9553283a (DEFAULT_TABLE table=test-table [id=01032cf9058844c1a167e0049f45298c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:33.771334 27223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1877a476d9594719ab8bcc8c9553283a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:33.771267 27549 tablet_service.cc:1467] Processing CreateTablet for tablet 1877a476d9594719ab8bcc8c9553283a (DEFAULT_TABLE table=test-table [id=01032cf9058844c1a167e0049f45298c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:33.771492 27389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1877a476d9594719ab8bcc8c9553283a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:33.771687 27549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1877a476d9594719ab8bcc8c9553283a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:33.777379 27772 tablet_bootstrap.cc:492] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4: Bootstrap starting.
I20260807 11:29:33.778494 27773 tablet_bootstrap.cc:492] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159: Bootstrap starting.
I20260807 11:29:33.779866 27772 tablet_bootstrap.cc:654] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:33.780403 27773 tablet_bootstrap.cc:654] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:33.782619 27772 tablet_bootstrap.cc:492] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4: No bootstrap required, opened a new log
I20260807 11:29:33.782822 27772 ts_tablet_manager.cc:1397] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260807 11:29:33.783957 27772 raft_consensus.cc:348] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:33.784310 27772 raft_consensus.cc:374] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:33.784484 27772 raft_consensus.cc:729] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48f3035cbf447eb9e1b6a9f97dc3ae4, State: Initialized, Role: FOLLOWER
I20260807 11:29:33.784834 27772 consensus_queue.cc:260] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:33.785969 27772 ts_tablet_manager.cc:1428] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260807 11:29:33.786252 27775 tablet_bootstrap.cc:492] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581: Bootstrap starting.
I20260807 11:29:33.788290 27775 tablet_bootstrap.cc:654] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:33.788841 27773 tablet_bootstrap.cc:492] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159: No bootstrap required, opened a new log
I20260807 11:29:33.789070 27773 ts_tablet_manager.cc:1397] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20260807 11:29:33.790036 27773 raft_consensus.cc:348] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:33.790331 27773 raft_consensus.cc:374] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:33.790449 27773 raft_consensus.cc:729] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48be9ef638fa43e390b78079c00d0159, State: Initialized, Role: FOLLOWER
I20260807 11:29:33.790654 27775 tablet_bootstrap.cc:492] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581: No bootstrap required, opened a new log
I20260807 11:29:33.790769 27773 consensus_queue.cc:260] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:33.790890 27775 ts_tablet_manager.cc:1397] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260807 11:29:33.791400 27773 ts_tablet_manager.cc:1428] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260807 11:29:33.791694 27775 raft_consensus.cc:348] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:33.791936 27775 raft_consensus.cc:374] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:33.792061 27775 raft_consensus.cc:729] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4837e34e7d4c4af9b0df7a291c67e581, State: Initialized, Role: FOLLOWER
I20260807 11:29:33.792320 27775 consensus_queue.cc:260] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:33.792881 27775 ts_tablet_manager.cc:1428] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 11:29:33.844304 27778 raft_consensus.cc:482] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:29:33.844599 27778 raft_consensus.cc:504] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:33.845495 27778 leader_election.cc:288] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821), 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597)
I20260807 11:29:33.846287 27570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1877a476d9594719ab8bcc8c9553283a" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48be9ef638fa43e390b78079c00d0159" is_pre_election: true
I20260807 11:29:33.846287 27244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1877a476d9594719ab8bcc8c9553283a" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4837e34e7d4c4af9b0df7a291c67e581" is_pre_election: true
I20260807 11:29:33.846565 27244 raft_consensus.cc:2454] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 0.
I20260807 11:29:33.846565 27570 raft_consensus.cc:2454] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 0.
I20260807 11:29:33.847158 27327 leader_election.cc:302] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48be9ef638fa43e390b78079c00d0159, f48f3035cbf447eb9e1b6a9f97dc3ae4; no voters:
I20260807 11:29:33.847470 27778 raft_consensus.cc:2781] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:29:33.847605 27778 raft_consensus.cc:482] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:29:33.847703 27778 raft_consensus.cc:3037] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:33.849856 27778 raft_consensus.cc:504] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:33.850390 27778 leader_election.cc:288] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 election: Requested vote from peers 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821), 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597)
I20260807 11:29:33.850908 27244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1877a476d9594719ab8bcc8c9553283a" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4837e34e7d4c4af9b0df7a291c67e581"
I20260807 11:29:33.850982 27570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1877a476d9594719ab8bcc8c9553283a" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48be9ef638fa43e390b78079c00d0159"
I20260807 11:29:33.851176 27244 raft_consensus.cc:3037] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:33.851325 27570 raft_consensus.cc:3037] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:33.854255 27244 raft_consensus.cc:2454] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 1.
I20260807 11:29:33.854439 27570 raft_consensus.cc:2454] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 1.
I20260807 11:29:33.854741 27327 leader_election.cc:302] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4837e34e7d4c4af9b0df7a291c67e581, f48f3035cbf447eb9e1b6a9f97dc3ae4; no voters:
I20260807 11:29:33.855067 27778 raft_consensus.cc:2781] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:33.855640 27778 raft_consensus.cc:686] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 LEADER]: Becoming Leader. State: Replica: f48f3035cbf447eb9e1b6a9f97dc3ae4, State: Running, Role: LEADER
I20260807 11:29:33.855942 27778 consensus_queue.cc:237] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:33.858798 27057 catalog_manager.cc:5495] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66). New cstate: current_term: 1 leader_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } health_report { overall_health: UNKNOWN } } }
I20260807 11:29:33.892777 27784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:33.887234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.24.103.66:36761)
0807 11:29:33.887932 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:33.887936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:33.887954 (+ 18us) server_negotiation.cc:407] Connection header received
0807 11:29:33.888006 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:33.888010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:33.888073 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:33.888185 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:33.888607 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:33.889373 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:33.890671 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:33.891171 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:33.891310 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:33.891899 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:33.892054 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:33.892136 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:33.892316 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 11:29:33.892383 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":355,"threads_started":1}
I20260807 11:29:33.903159 27570 raft_consensus.cc:1261] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Refusing update from remote peer f48f3035cbf447eb9e1b6a9f97dc3ae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:33.903262 27244 raft_consensus.cc:1261] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Refusing update from remote peer f48f3035cbf447eb9e1b6a9f97dc3ae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:33.903743 27778 consensus_queue.cc:1035] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:33.904309 27778 consensus_queue.cc:1035] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:33.956079 27056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:33.956444 27056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:33.960467 27056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b: Sending DeleteTablet for 3 replicas of tablet 1877a476d9594719ab8bcc8c9553283a
I20260807 11:29:33.961560 27223 tablet_service.cc:1514] Processing DeleteTablet for tablet 1877a476d9594719ab8bcc8c9553283a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:33 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:56674
I20260807 11:29:33.961597 27549 tablet_service.cc:1514] Processing DeleteTablet for tablet 1877a476d9594719ab8bcc8c9553283a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:33 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:52950
I20260807 11:29:33.961902 27763 tablet_replica.cc:330] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159: stopping tablet replica
I20260807 11:29:33.961938 27762 tablet_replica.cc:330] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581: stopping tablet replica
I20260807 11:29:33.962322 27762 raft_consensus.cc:2229] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:33.962376 27763 raft_consensus.cc:2229] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:33.962543 27762 raft_consensus.cc:2258] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:33.962651 27763 raft_consensus.cc:2258] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:33.962852 27389 tablet_service.cc:1514] Processing DeleteTablet for tablet 1877a476d9594719ab8bcc8c9553283a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:33 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:45180
I20260807 11:29:33.963223 27761 tablet_replica.cc:330] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4: stopping tablet replica
I20260807 11:29:33.963202 27056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 11:29:33.963490 27761 raft_consensus.cc:2229] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:29:33.963860 27761 raft_consensus.cc:2258] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:33.964107 27762 ts_tablet_manager.cc:1905] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 11:29:33.964282 27056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:29:33.964391 27763 ts_tablet_manager.cc:1905] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:33.966948 27761 ts_tablet_manager.cc:1905] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:33.972285 27762 ts_tablet_manager.cc:1918] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:29:33.972512 27763 ts_tablet_manager.cc:1918] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:29:33.972595 27762 log.cc:1196] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/1877a476d9594719ab8bcc8c9553283a
I20260807 11:29:33.973080 27762 ts_tablet_manager.cc:1939] T 1877a476d9594719ab8bcc8c9553283a P 4837e34e7d4c4af9b0df7a291c67e581: Deleting consensus metadata
I20260807 11:29:33.973258 27763 log.cc:1196] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/1877a476d9594719ab8bcc8c9553283a
I20260807 11:29:33.973824 27763 ts_tablet_manager.cc:1939] T 1877a476d9594719ab8bcc8c9553283a P 48be9ef638fa43e390b78079c00d0159: Deleting consensus metadata
I20260807 11:29:33.974443 27041 catalog_manager.cc:4841] TS 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821): tablet 1877a476d9594719ab8bcc8c9553283a (table test-table [id=01032cf9058844c1a167e0049f45298c]) successfully deleted
I20260807 11:29:33.975323 27041 catalog_manager.cc:4841] TS 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597): tablet 1877a476d9594719ab8bcc8c9553283a (table test-table [id=01032cf9058844c1a167e0049f45298c]) successfully deleted
I20260807 11:29:33.976115 27761 ts_tablet_manager.cc:1918] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:29:33.976310 27761 log.cc:1196] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/1877a476d9594719ab8bcc8c9553283a
I20260807 11:29:33.976847 27761 ts_tablet_manager.cc:1939] T 1877a476d9594719ab8bcc8c9553283a P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting consensus metadata
I20260807 11:29:33.978209 27043 catalog_manager.cc:4841] TS f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66:36761): tablet 1877a476d9594719ab8bcc8c9553283a (table test-table [id=01032cf9058844c1a167e0049f45298c]) successfully deleted
I20260807 11:29:33.979781 27389 tablet_service.cc:1467] Processing CreateTablet for tablet 023176464db24b09aef594ce3b8dab1c (DEFAULT_TABLE table=test-table [id=df638b053a0c410ea92d565a06a0f5e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:33.979796 27223 tablet_service.cc:1467] Processing CreateTablet for tablet 023176464db24b09aef594ce3b8dab1c (DEFAULT_TABLE table=test-table [id=df638b053a0c410ea92d565a06a0f5e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:33.980312 27389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023176464db24b09aef594ce3b8dab1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:33.980333 27223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023176464db24b09aef594ce3b8dab1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:33.983109 27549 tablet_service.cc:1467] Processing CreateTablet for tablet 023176464db24b09aef594ce3b8dab1c (DEFAULT_TABLE table=test-table [id=df638b053a0c410ea92d565a06a0f5e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:33.983669 27549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023176464db24b09aef594ce3b8dab1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:33.985630 27772 tablet_bootstrap.cc:492] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Bootstrap starting.
I20260807 11:29:33.987529 27772 tablet_bootstrap.cc:654] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:33.987592 27775 tablet_bootstrap.cc:492] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581: Bootstrap starting.
I20260807 11:29:33.989768 27775 tablet_bootstrap.cc:654] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:33.991968 27773 tablet_bootstrap.cc:492] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159: Bootstrap starting.
I20260807 11:29:33.993726 27773 tablet_bootstrap.cc:654] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:33.998327 27775 tablet_bootstrap.cc:492] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581: No bootstrap required, opened a new log
I20260807 11:29:33.998934 27775 ts_tablet_manager.cc:1397] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260807 11:29:33.999852 27775 raft_consensus.cc:348] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:34.000200 27775 raft_consensus.cc:374] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:34.000346 27775 raft_consensus.cc:729] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4837e34e7d4c4af9b0df7a291c67e581, State: Initialized, Role: FOLLOWER
I20260807 11:29:34.000763 27773 tablet_bootstrap.cc:492] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159: No bootstrap required, opened a new log
I20260807 11:29:34.000679 27775 consensus_queue.cc:260] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:34.001000 27773 ts_tablet_manager.cc:1397] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260807 11:29:34.002067 27773 raft_consensus.cc:348] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:34.002439 27773 raft_consensus.cc:374] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:34.002571 27773 raft_consensus.cc:729] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48be9ef638fa43e390b78079c00d0159, State: Initialized, Role: FOLLOWER
I20260807 11:29:34.002900 27773 consensus_queue.cc:260] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:34.004177 27775 ts_tablet_manager.cc:1428] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260807 11:29:34.005918 27773 ts_tablet_manager.cc:1428] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260807 11:29:34.007920 27772 tablet_bootstrap.cc:492] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4: No bootstrap required, opened a new log
I20260807 11:29:34.008106 27772 ts_tablet_manager.cc:1397] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.004s
I20260807 11:29:34.008868 27772 raft_consensus.cc:348] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:34.009133 27772 raft_consensus.cc:374] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:34.009204 27772 raft_consensus.cc:729] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48f3035cbf447eb9e1b6a9f97dc3ae4, State: Initialized, Role: FOLLOWER
I20260807 11:29:34.009454 27772 consensus_queue.cc:260] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:34.009977 27772 ts_tablet_manager.cc:1428] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260807 11:29:34.029771 27778 raft_consensus.cc:482] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:29:34.030000 27778 raft_consensus.cc:504] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:34.030782 27778 leader_election.cc:288] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821), 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597)
I20260807 11:29:34.031461 27244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023176464db24b09aef594ce3b8dab1c" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4837e34e7d4c4af9b0df7a291c67e581" is_pre_election: true
I20260807 11:29:34.031473 27570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023176464db24b09aef594ce3b8dab1c" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48be9ef638fa43e390b78079c00d0159" is_pre_election: true
I20260807 11:29:34.031723 27244 raft_consensus.cc:2454] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 0.
I20260807 11:29:34.031771 27570 raft_consensus.cc:2454] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 0.
I20260807 11:29:34.032189 27327 leader_election.cc:302] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4837e34e7d4c4af9b0df7a291c67e581, f48f3035cbf447eb9e1b6a9f97dc3ae4; no voters:
I20260807 11:29:34.032536 27778 raft_consensus.cc:2781] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:29:34.032665 27778 raft_consensus.cc:482] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:29:34.032817 27778 raft_consensus.cc:3037] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:34.035207 27778 raft_consensus.cc:504] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:34.035832 27778 leader_election.cc:288] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 election: Requested vote from peers 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821), 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597)
I20260807 11:29:34.036437 27244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023176464db24b09aef594ce3b8dab1c" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4837e34e7d4c4af9b0df7a291c67e581"
I20260807 11:29:34.036696 27244 raft_consensus.cc:3037] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:34.036576 27570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023176464db24b09aef594ce3b8dab1c" candidate_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48be9ef638fa43e390b78079c00d0159"
I20260807 11:29:34.036870 27570 raft_consensus.cc:3037] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:34.039474 27570 raft_consensus.cc:2454] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 1.
I20260807 11:29:34.039577 27244 raft_consensus.cc:2454] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48f3035cbf447eb9e1b6a9f97dc3ae4 in term 1.
I20260807 11:29:34.039919 27327 leader_election.cc:302] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48be9ef638fa43e390b78079c00d0159, f48f3035cbf447eb9e1b6a9f97dc3ae4; no voters:
I20260807 11:29:34.040221 27778 raft_consensus.cc:2781] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:34.040441 27778 raft_consensus.cc:686] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 LEADER]: Becoming Leader. State: Replica: f48f3035cbf447eb9e1b6a9f97dc3ae4, State: Running, Role: LEADER
I20260807 11:29:34.040805 27778 consensus_queue.cc:237] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } }
I20260807 11:29:34.043910 27057 catalog_manager.cc:5495] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66). New cstate: current_term: 1 leader_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f48f3035cbf447eb9e1b6a9f97dc3ae4" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 } health_report { overall_health: UNKNOWN } } }
I20260807 11:29:34.074168 27570 raft_consensus.cc:1261] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Refusing update from remote peer f48f3035cbf447eb9e1b6a9f97dc3ae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:34.074343 27244 raft_consensus.cc:1261] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Refusing update from remote peer f48f3035cbf447eb9e1b6a9f97dc3ae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:34.074908 27789 consensus_queue.cc:1035] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48be9ef638fa43e390b78079c00d0159" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:34.075286 27778 consensus_queue.cc:1035] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4837e34e7d4c4af9b0df7a291c67e581" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 11:29:34.096707 27742 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 11:29:35.688059 27057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:35.688316 27057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:35.693058 27057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6cdd1343b864a8fa79d6ccfb74fe34b: Sending DeleteTablet for 3 replicas of tablet 023176464db24b09aef594ce3b8dab1c
I20260807 11:29:35.694085 27223 tablet_service.cc:1514] Processing DeleteTablet for tablet 023176464db24b09aef594ce3b8dab1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:35 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:56674
I20260807 11:29:35.694142 27389 tablet_service.cc:1514] Processing DeleteTablet for tablet 023176464db24b09aef594ce3b8dab1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:35 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:45180
I20260807 11:29:35.694264 27549 tablet_service.cc:1514] Processing DeleteTablet for tablet 023176464db24b09aef594ce3b8dab1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:35 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:52950
I20260807 11:29:35.695143 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 27144
I20260807 11:29:35.706964 27827 tablet_replica.cc:330] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581: stopping tablet replica
I20260807 11:29:35.707532 27827 raft_consensus.cc:2229] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:35.708027 27827 raft_consensus.cc:2258] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:35.708796 27829 tablet_replica.cc:330] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159: stopping tablet replica
I20260807 11:29:35.709750 27829 raft_consensus.cc:2229] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:35.709779 27827 ts_tablet_manager.cc:1905] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:35.710160 27829 raft_consensus.cc:2258] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:35.710808 27828 tablet_replica.cc:330] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4: stopping tablet replica
I20260807 11:29:35.711109 27828 raft_consensus.cc:2229] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:29:35.711601 27828 raft_consensus.cc:2258] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:35.712268 27829 ts_tablet_manager.cc:1905] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:35.712708 27828 ts_tablet_manager.cc:1905] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:35.717945 27827 ts_tablet_manager.cc:1918] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:29:35.718186 27827 log.cc:1196] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/023176464db24b09aef594ce3b8dab1c
I20260807 11:29:35.718883 27827 ts_tablet_manager.cc:1939] T 023176464db24b09aef594ce3b8dab1c P 4837e34e7d4c4af9b0df7a291c67e581: Deleting consensus metadata
I20260807 11:29:35.719836 27829 ts_tablet_manager.cc:1918] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:29:35.719859 27828 ts_tablet_manager.cc:1918] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 11:29:35.719988 27828 log.cc:1196] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/023176464db24b09aef594ce3b8dab1c
I20260807 11:29:35.720075 27829 log.cc:1196] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/023176464db24b09aef594ce3b8dab1c
I20260807 11:29:35.720568 27828 ts_tablet_manager.cc:1939] T 023176464db24b09aef594ce3b8dab1c P f48f3035cbf447eb9e1b6a9f97dc3ae4: Deleting consensus metadata
I20260807 11:29:35.720613 27829 ts_tablet_manager.cc:1939] T 023176464db24b09aef594ce3b8dab1c P 48be9ef638fa43e390b78079c00d0159: Deleting consensus metadata
I20260807 11:29:35.720587 27041 catalog_manager.cc:4841] TS 4837e34e7d4c4af9b0df7a291c67e581 (127.24.103.65:37821): tablet 023176464db24b09aef594ce3b8dab1c (table test-table [id=df638b053a0c410ea92d565a06a0f5e0]) successfully deleted
Aug 07 11:29:35 dist-test-slave-wl2h krb5kdc[27005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:29:35.721819 27041 catalog_manager.cc:4841] TS 48be9ef638fa43e390b78079c00d0159 (127.24.103.67:34597): tablet 023176464db24b09aef594ce3b8dab1c (table test-table [id=df638b053a0c410ea92d565a06a0f5e0]) successfully deleted
I20260807 11:29:35.721855 27043 catalog_manager.cc:4841] TS f48f3035cbf447eb9e1b6a9f97dc3ae4 (127.24.103.66:36761): tablet 023176464db24b09aef594ce3b8dab1c (table test-table [id=df638b053a0c410ea92d565a06a0f5e0]) successfully deleted
I20260807 11:29:35.728088 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:35.697566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.24.103.65:37821)
0807 11:29:35.709737 (+ 12171us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:35.709745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:35.709788 (+ 43us) server_negotiation.cc:407] Connection header received
0807 11:29:35.709877 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:35.709886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:35.710001 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:35.710159 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:35.712146 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:35.713350 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:35.715229 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:35.715934 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:35.716034 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:35.721925 (+ 5891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:35.721949 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:35.721970 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:35.722014 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:35.725145 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:35.725819 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:35.725827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:35.725829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:35.725962 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:35.726441 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:35.726444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:35.726446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:35.726809 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:35.727116 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:35.727465 (+ 349us) server_negotiation.cc:299] Negotiation successful
0807 11:29:35.727737 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11985,"thread_start_us":11867,"threads_started":1}
W20260807 11:29:35.728806 27296 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54944
W20260807 11:29:35.729532 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 27144: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:29:35.729743 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 27144
I20260807 11:29:35.757553 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 27306
Aug 07 11:29:35 dist-test-slave-wl2h krb5kdc[27005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:29:35.773708 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:35.758324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.24.103.66:36761)
0807 11:29:35.758913 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:35.758917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:35.758934 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:29:35.758991 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:35.758995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:35.759112 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:35.759215 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:35.760523 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:35.761280 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:35.762616 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:35.763105 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:35.763204 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:35.767673 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:35.767688 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:35.767704 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:35.767749 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:35.770944 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:35.771775 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:35.771784 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:35.771786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:35.771926 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:35.772284 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:35.772288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:35.772290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:35.772702 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:35.772940 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:35.773221 (+ 281us) server_negotiation.cc:299] Negotiation successful
0807 11:29:35.773411 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":313,"threads_started":1}
W20260807 11:29:35.774562 27462 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45294
W20260807 11:29:35.775513 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 27306: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:29:35.775684 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 27306
I20260807 11:29:35.805716 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 27473
Aug 07 11:29:35 dist-test-slave-wl2h krb5kdc[27005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:29:35.823541 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:35.806573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.24.103.67:34597)
0807 11:29:35.807250 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:35.807255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:35.807274 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:29:35.807336 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:35.807340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:35.807406 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:35.807513 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:35.808984 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:35.809892 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:35.811347 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:35.811766 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:35.811885 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:35.817141 (+ 5256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:35.817158 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:35.817176 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:35.817218 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:35.820430 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:35.821260 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:35.821268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:35.821270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:35.821417 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:35.821852 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:35.821856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:35.821858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:35.822302 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:35.822532 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:35.822813 (+ 281us) server_negotiation.cc:299] Negotiation successful
0807 11:29:35.822997 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":323,"threads_started":1}
W20260807 11:29:35.824342 27621 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40690
W20260807 11:29:35.825038 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 27473: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:29:35.825237 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 27473
I20260807 11:29:35.852516 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 27022
I20260807 11:29:35.853861 27108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:35.973943 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 27022
2026-08-07T11:29:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8664 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:29:36 dist-test-slave-wl2h krb5kdc[27837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:29:36 dist-test-slave-wl2h krb5kdc[27837](info): set up 2 sockets
Aug 07 11:29:36 dist-test-slave-wl2h krb5kdc[27837](info): commencing operation
krb5kdc: starting...
W20260807 11:29:37.219602 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:29:37 dist-test-slave-wl2h krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:29:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:29:37.471994 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41239
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:37.809062 27854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:37.809471 27854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:37.809630 27854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:37.817293 27854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:37.817469 27854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:37.817574 27854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:37.817680 27854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:37.817759 27854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:29:37.828408 27854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:37.829887 27854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:37.831324 27854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:37.844555 27860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:37.844521 27859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:37.844611 27862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:37.845386 27854 server_base.cc:1034] running on GCE node
I20260807 11:29:37.846290 27854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:37.847899 27854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:37.849175 27854 hybrid_clock.cc:648] HybridClock initialized: now 1786102177849121 us; error 60 us; skew 500 ppm
I20260807 11:29:37.849558 27854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:37 dist-test-slave-wl2h krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102177, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:37.857798 27854 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:29:37.859687 27854 webserver.cc:460] Webserver started at http://127.24.103.126:34037/ using document root <none> and password file <none>
I20260807 11:29:37.860464 27854 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:37.860632 27854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:37.860980 27854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:37.864341 27854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "6943726b088849459aaf8fc99268d1e1"
format_stamp: "Formatted at 2026-08-07 11:29:37 on dist-test-slave-wl2h"
server_key: "58f7c85e6e1cd10a516558a5f194d7fd"
server_key_iv: "ee6edd144a0ab910214e5a02c2b8859f"
server_key_version: "encryptionkey@0"
I20260807 11:29:37.865177 27854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "6943726b088849459aaf8fc99268d1e1"
format_stamp: "Formatted at 2026-08-07 11:29:37 on dist-test-slave-wl2h"
server_key: "58f7c85e6e1cd10a516558a5f194d7fd"
server_key_iv: "ee6edd144a0ab910214e5a02c2b8859f"
server_key_version: "encryptionkey@0"
I20260807 11:29:37.870712 27854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 11:29:37.874435 27870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:37.876084 27854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:37.876286 27854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "6943726b088849459aaf8fc99268d1e1"
format_stamp: "Formatted at 2026-08-07 11:29:37 on dist-test-slave-wl2h"
server_key: "58f7c85e6e1cd10a516558a5f194d7fd"
server_key_iv: "ee6edd144a0ab910214e5a02c2b8859f"
server_key_version: "encryptionkey@0"
I20260807 11:29:37.876514 27854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:37.911482 27854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:37.916297 27854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:37.916675 27854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:37.938095 27854 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:41239
I20260807 11:29:37.938133 27934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:41239 every 8 connection(s)
I20260807 11:29:37.939653 27854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:37.943521 27935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:37.946964 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 27854
I20260807 11:29:37.947242 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:37.948103 24989 external_mini_cluster.cc:1428] Setting key 72dde2744436fb207b4f728fdbbefdd7
I20260807 11:29:37.957454 27935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1: Bootstrap starting.
I20260807 11:29:37.961404 27935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:37.962769 27935 log.cc:824] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1: Log is configured to *not* fsync() on all Append() calls
Aug 07 11:29:37 dist-test-slave-wl2h krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:37.966513 27935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1: No bootstrap required, opened a new log
I20260807 11:29:37.974115 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:37.950366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.24.103.126:41239)
0807 11:29:37.951189 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:37.951200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:37.951252 (+ 52us) server_negotiation.cc:407] Connection header received
0807 11:29:37.953054 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:37.953078 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:37.953518 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:37.953885 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:37.955878 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:37.957285 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:37.959127 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:37.959882 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:37.960022 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:37.965492 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:37.965501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:37.965515 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:37.965562 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:37.969192 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:37.970146 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:37.970153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:37.970160 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:37.970350 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:37.970841 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:37.970845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:37.970847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:37.971534 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:37.971799 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:37.972367 (+ 568us) server_negotiation.cc:299] Negotiation successful
0807 11:29:37.972970 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":378,"threads_started":1}
I20260807 11:29:37.976274 27935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6943726b088849459aaf8fc99268d1e1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41239 } }
I20260807 11:29:37.976692 27935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:37.976823 27935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6943726b088849459aaf8fc99268d1e1, State: Initialized, Role: FOLLOWER
I20260807 11:29:37.977485 27935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6943726b088849459aaf8fc99268d1e1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41239 } }
I20260807 11:29:37.977766 27935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:37.977918 27935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:37.978108 27935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:37.980548 27935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6943726b088849459aaf8fc99268d1e1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41239 } }
I20260807 11:29:37.981240 27935 leader_election.cc:302] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6943726b088849459aaf8fc99268d1e1; no voters:
I20260807 11:29:37.981973 27935 leader_election.cc:288] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:37.982182 27941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:37.983422 27941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [term 1 LEADER]: Becoming Leader. State: Replica: 6943726b088849459aaf8fc99268d1e1, State: Running, Role: LEADER
I20260807 11:29:37.983933 27941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6943726b088849459aaf8fc99268d1e1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41239 } }
I20260807 11:29:37.984305 27935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:37.987052 27942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6943726b088849459aaf8fc99268d1e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6943726b088849459aaf8fc99268d1e1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41239 } } }
I20260807 11:29:37.986981 27943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6943726b088849459aaf8fc99268d1e1. Latest consensus state: current_term: 1 leader_uuid: "6943726b088849459aaf8fc99268d1e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6943726b088849459aaf8fc99268d1e1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41239 } } }
I20260807 11:29:37.987452 27943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:37.987452 27942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1 [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:37.988971 27951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:37.993652 27951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:38.002616 27951 catalog_manager.cc:1347] Generated new cluster ID: 77c73d5ba518479b8720f5c1229d951d
I20260807 11:29:38.002754 27951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:38.011633 27951 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:38.013167 27951 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:38.025444 27951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6943726b088849459aaf8fc99268d1e1: Generated new TSK 0
I20260807 11:29:38.026343 27951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:29:38.095496 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41239
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:38.451615 27965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:38.451898 27965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:38.452088 27965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:38.459426 27965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:38.459542 27965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:38.459753 27965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:29:38.469692 27965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:38.471129 27965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:38.472522 27965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:38.486615 27970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:38.486620 27971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:38.486824 27973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:38.487850 27965 server_base.cc:1034] running on GCE node
I20260807 11:29:38.488672 27965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:38.489661 27965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:38.490871 27965 hybrid_clock.cc:648] HybridClock initialized: now 1786102178490824 us; error 35 us; skew 500 ppm
I20260807 11:29:38.491266 27965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:38 dist-test-slave-wl2h krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102178, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:38.494501 27965 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:29:38.496150 27965 webserver.cc:460] Webserver started at http://127.24.103.65:42191/ using document root <none> and password file <none>
I20260807 11:29:38.496866 27965 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:38.497018 27965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:38.497340 27965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:38.500566 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "fbd0180583c549d0b40a335ce13d5e26"
format_stamp: "Formatted at 2026-08-07 11:29:38 on dist-test-slave-wl2h"
server_key: "d40df6032ece19c5b4e88aa3f4bcc8a1"
server_key_iv: "e663855c4363d7394d4dfdf8a6375662"
server_key_version: "encryptionkey@0"
I20260807 11:29:38.501413 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "fbd0180583c549d0b40a335ce13d5e26"
format_stamp: "Formatted at 2026-08-07 11:29:38 on dist-test-slave-wl2h"
server_key: "d40df6032ece19c5b4e88aa3f4bcc8a1"
server_key_iv: "e663855c4363d7394d4dfdf8a6375662"
server_key_version: "encryptionkey@0"
I20260807 11:29:38.506748 27965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260807 11:29:38.510514 27981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:38.512265 27965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:29:38.512475 27965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "fbd0180583c549d0b40a335ce13d5e26"
format_stamp: "Formatted at 2026-08-07 11:29:38 on dist-test-slave-wl2h"
server_key: "d40df6032ece19c5b4e88aa3f4bcc8a1"
server_key_iv: "e663855c4363d7394d4dfdf8a6375662"
server_key_version: "encryptionkey@0"
I20260807 11:29:38.512712 27965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:38.538527 27965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:38.543262 27965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:38.543596 27965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:38.544678 27965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:38.546140 27965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:38.546269 27965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:38.546440 27965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:38.546521 27965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:38.584414 27965 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:35121
I20260807 11:29:38.584491 28114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:35121 every 8 connection(s)
I20260807 11:29:38.586058 27965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:29:38.590957 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 27965
I20260807 11:29:38.591248 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:29:38.591897 24989 external_mini_cluster.cc:1428] Setting key fe27dc2904e433ef9ec2a089de96e28b
Aug 07 11:29:38 dist-test-slave-wl2h krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102178, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:38.608042 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:38.588781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:44471 (local address 127.24.103.126:41239)
0807 11:29:38.589293 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:38.589299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:38.590285 (+ 986us) server_negotiation.cc:407] Connection header received
0807 11:29:38.592335 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:38.592341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:38.592463 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:38.592589 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:38.594363 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:38.595550 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:38.597104 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:38.597857 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:38.598005 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:38.601889 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:38.601894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:38.601897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:38.601930 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:38.604619 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:38.605586 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:38.605590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:38.605592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:38.605709 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:38.606195 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:38.606198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:38.606200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:38.606463 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:38.606575 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:38.607609 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0807 11:29:38.607785 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":243,"threads_started":1}
I20260807 11:29:38.609618 28118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:38.589389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41239 (local address 127.24.103.65:44471)
0807 11:29:38.590109 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:38.590156 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:38.592057 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:38.592896 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:38.592908 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:38.593478 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:38.594181 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:38.594199 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:38.595727 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:38.595732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:38.596950 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:38.596962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:38.598076 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:38.598080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:38.598286 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:38.599131 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:38.599154 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:38.601665 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:38.604763 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:38.604775 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:38.604795 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:38.605471 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:38.605830 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:38.605837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:38.605844 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:38.606083 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:38.606601 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:38.606612 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:38.607161 (+ 549us) client_negotiation.cc:769] Sending connection context
0807 11:29:38.607495 (+ 334us) client_negotiation.cc:240] Negotiation successful
0807 11:29:38.608073 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":279,"threads_started":1}
I20260807 11:29:38.611331 28115 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41239
I20260807 11:29:38.611667 28115 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:38.612418 28115 heartbeater.cc:511] Master 127.24.103.126:41239 requested a full tablet report, sending...
I20260807 11:29:38.614555 27892 ts_manager.cc:194] Registered new tserver with Master: fbd0180583c549d0b40a335ce13d5e26 (127.24.103.65:35121)
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260807 11:29:38.616187 27892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:44471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:29:38.647529 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41239
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:38.984000 28121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:38.984292 28121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:38.984390 28121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:38.991863 28121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:38.991986 28121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:38.992125 28121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:29:39.002307 28121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:39.003620 28121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:39.004899 28121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:39.019917 28126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:39.019898 28127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:39.020164 28130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:39.020810 28121 server_base.cc:1034] running on GCE node
I20260807 11:29:39.021469 28121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:39.022549 28121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:39.023775 28121 hybrid_clock.cc:648] HybridClock initialized: now 1786102179023720 us; error 67 us; skew 500 ppm
I20260807 11:29:39.024140 28121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:39 dist-test-slave-wl2h krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102179, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:39.027930 28121 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:29:39.029584 28121 webserver.cc:460] Webserver started at http://127.24.103.66:36985/ using document root <none> and password file <none>
I20260807 11:29:39.030331 28121 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:39.030485 28121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:39.030822 28121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:39.034008 28121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "b7fe05fc31514ff69695440172a64a77"
format_stamp: "Formatted at 2026-08-07 11:29:39 on dist-test-slave-wl2h"
server_key: "42ca8d560a0faa2756288467ca1d6e81"
server_key_iv: "26ae217e8e2510aa1d37ee80c4c93364"
server_key_version: "encryptionkey@0"
I20260807 11:29:39.034879 28121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "b7fe05fc31514ff69695440172a64a77"
format_stamp: "Formatted at 2026-08-07 11:29:39 on dist-test-slave-wl2h"
server_key: "42ca8d560a0faa2756288467ca1d6e81"
server_key_iv: "26ae217e8e2510aa1d37ee80c4c93364"
server_key_version: "encryptionkey@0"
I20260807 11:29:39.040544 28121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 11:29:39.043972 28138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:39.045329 28121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:29:39.045466 28121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "b7fe05fc31514ff69695440172a64a77"
format_stamp: "Formatted at 2026-08-07 11:29:39 on dist-test-slave-wl2h"
server_key: "42ca8d560a0faa2756288467ca1d6e81"
server_key_iv: "26ae217e8e2510aa1d37ee80c4c93364"
server_key_version: "encryptionkey@0"
I20260807 11:29:39.045650 28121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:39.078455 28121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:39.082861 28121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:39.083134 28121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:39.084182 28121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:39.085631 28121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:39.085705 28121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:39.085805 28121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:39.085858 28121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:39.120859 28121 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:32901
I20260807 11:29:39.120950 28269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:32901 every 8 connection(s)
I20260807 11:29:39.122159 28121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:29:39.131909 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 28121
I20260807 11:29:39.132175 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:29:39.132697 24989 external_mini_cluster.cc:1428] Setting key 68e0a77c2025800d7c02ae4de03744ab
Aug 07 11:29:39 dist-test-slave-wl2h krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102179, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:39.142977 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:39.124932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:60739 (local address 127.24.103.126:41239)
0807 11:29:39.125414 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:39.125418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:39.126219 (+ 801us) server_negotiation.cc:407] Connection header received
0807 11:29:39.128208 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:39.128213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:39.128310 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:39.128426 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:39.129892 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:39.130626 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:39.132284 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:39.132723 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:39.132875 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:39.137118 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:39.137123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:39.137126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:39.137158 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:39.139885 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:39.140636 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:39.140641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:39.140643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:39.140775 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:39.141225 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:39.141234 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:39.141237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:39.141503 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:39.141663 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:39.142349 (+ 686us) server_negotiation.cc:299] Negotiation successful
0807 11:29:39.142584 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":279,"threads_started":1}
I20260807 11:29:39.143986 28274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:39.125349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41239 (local address 127.24.103.66:60739)
0807 11:29:39.126053 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:39.126090 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:39.127933 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:39.128663 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:39.128671 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:39.129183 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:39.129716 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:39.129732 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:39.130798 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:39.130802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:39.132116 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:39.132130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:39.132967 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:39.132975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:39.133218 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:39.134094 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:39.134122 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:39.136878 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:39.140037 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:39.140049 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:39.140065 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:39.140498 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:39.140889 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:39.140894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:39.140898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:39.141101 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:39.141637 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:39.141645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:39.141994 (+ 349us) client_negotiation.cc:769] Sending connection context
0807 11:29:39.142272 (+ 278us) client_negotiation.cc:240] Negotiation successful
0807 11:29:39.142809 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":300,"threads_started":1}
I20260807 11:29:39.145527 28270 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41239
I20260807 11:29:39.145843 28270 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:39.146543 28270 heartbeater.cc:511] Master 127.24.103.126:41239 requested a full tablet report, sending...
I20260807 11:29:39.148135 27892 ts_manager.cc:194] Registered new tserver with Master: b7fe05fc31514ff69695440172a64a77 (127.24.103.66:32901)
I20260807 11:29:39.149415 27892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:60739
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:29:39.187155 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41239
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:39.533789 28277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:39.534111 28277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:39.534224 28277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:39.541633 28277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:39.541754 28277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:39.541867 28277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:29:39.552349 28277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:39.553663 28277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:39.554958 28277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:39.568729 28284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:39.568677 28287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:39.568648 28283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:39.569856 28277 server_base.cc:1034] running on GCE node
I20260807 11:29:39.570468 28277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:39.571470 28277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:39.572681 28277 hybrid_clock.cc:648] HybridClock initialized: now 1786102179572633 us; error 35 us; skew 500 ppm
I20260807 11:29:39.573012 28277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:39 dist-test-slave-wl2h krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102179, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:39.576210 28277 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:29:39.577800 28277 webserver.cc:460] Webserver started at http://127.24.103.67:35497/ using document root <none> and password file <none>
I20260807 11:29:39.578516 28277 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:39.578670 28277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:39.578995 28277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:39.582235 28277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "5293615aa9d348ef89577f516e38dd63"
format_stamp: "Formatted at 2026-08-07 11:29:39 on dist-test-slave-wl2h"
server_key: "0bd9130d9b1caf1e5ef94c6b7ac4a3cd"
server_key_iv: "8138e0a50539173a8737ce852cf514ee"
server_key_version: "encryptionkey@0"
I20260807 11:29:39.583086 28277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "5293615aa9d348ef89577f516e38dd63"
format_stamp: "Formatted at 2026-08-07 11:29:39 on dist-test-slave-wl2h"
server_key: "0bd9130d9b1caf1e5ef94c6b7ac4a3cd"
server_key_iv: "8138e0a50539173a8737ce852cf514ee"
server_key_version: "encryptionkey@0"
I20260807 11:29:39.589063 28277 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260807 11:29:39.592784 28296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:39.594103 28277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260807 11:29:39.594308 28277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "5293615aa9d348ef89577f516e38dd63"
format_stamp: "Formatted at 2026-08-07 11:29:39 on dist-test-slave-wl2h"
server_key: "0bd9130d9b1caf1e5ef94c6b7ac4a3cd"
server_key_iv: "8138e0a50539173a8737ce852cf514ee"
server_key_version: "encryptionkey@0"
I20260807 11:29:39.594522 28277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:39.619509 28115 heartbeater.cc:503] Master 127.24.103.126:41239 was elected leader, sending a full tablet report...
I20260807 11:29:39.634668 28277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:39.639180 28277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:39.639467 28277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:39.640478 28277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:39.641916 28277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:39.642001 28277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:39.642087 28277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:39.642133 28277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:39.677620 28277 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:41215
I20260807 11:29:39.677716 28424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:41215 every 8 connection(s)
I20260807 11:29:39.679068 28277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:29:39.682113 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 28277
I20260807 11:29:39.682344 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:29:39.682906 24989 external_mini_cluster.cc:1428] Setting key 21f33927b136853474d3664150ee89e7
Aug 07 11:29:39 dist-test-slave-wl2h krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102179, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:39.701673 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:39.682098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:40015 (local address 127.24.103.126:41239)
0807 11:29:39.682688 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:39.682694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:39.686284 (+ 3590us) server_negotiation.cc:407] Connection header received
0807 11:29:39.686338 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:39.686342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:39.686408 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:39.686492 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:39.688346 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:39.689384 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:39.690992 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:39.691522 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:39.691636 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:39.695306 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:39.695312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:39.695315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:39.695349 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:39.698388 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:39.699271 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:39.699275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:39.699277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:39.699454 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:39.699935 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:39.699940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:39.699942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:39.700297 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:39.700441 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:39.701224 (+ 783us) server_negotiation.cc:299] Negotiation successful
0807 11:29:39.701395 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":282,"threads_started":1}
I20260807 11:29:39.702656 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:39.682760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41239 (local address 127.24.103.67:40015)
0807 11:29:39.683734 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:39.683793 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:39.685947 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:39.686749 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:39.686757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:39.687438 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:39.688122 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:39.688141 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:39.689559 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:39.689563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:39.690833 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:39.690845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:39.691777 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:39.691783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:39.691997 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:39.692803 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:39.692827 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:39.695093 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:39.698502 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:39.698512 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:39.698529 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:39.699163 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:39.699566 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:39.699569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:39.699572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:39.699756 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:39.700457 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:39.700463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:39.700808 (+ 345us) client_negotiation.cc:769] Sending connection context
0807 11:29:39.701056 (+ 248us) client_negotiation.cc:240] Negotiation successful
0807 11:29:39.701466 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":459,"threads_started":1}
I20260807 11:29:39.704152 28425 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41239
I20260807 11:29:39.704504 28425 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:39.705314 28425 heartbeater.cc:511] Master 127.24.103.126:41239 requested a full tablet report, sending...
I20260807 11:29:39.706887 27892 ts_manager.cc:194] Registered new tserver with Master: 5293615aa9d348ef89577f516e38dd63 (127.24.103.67:41215)
I20260807 11:29:39.707783 27892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:40015
I20260807 11:29:39.711478 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 11:29:39.754374 28427 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:29:39.751643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.24.103.126:41239)
0807 11:29:39.751879 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:39.751887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:39.752331 (+ 444us) server_negotiation.cc:407] Connection header received
0807 11:29:39.752560 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:39.752567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:39.752649 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:39.752769 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:39.754152 (+ 1383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44510: BlockingRecv error: recv got EOF from 127.0.0.1:44510 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:29:39.758327 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 27965
W20260807 11:29:39.761530 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 27965: Not authorized: Client connection negotiation failed: client connection to 127.24.103.65:35121: server requires authentication, but client does not have Kerberos credentials available
I20260807 11:29:39.761724 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 27965
I20260807 11:29:39.781328 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 28121
W20260807 11:29:39.784497 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 28121: Not authorized: Client connection negotiation failed: client connection to 127.24.103.66:32901: server requires authentication, but client does not have Kerberos credentials available
I20260807 11:29:39.784646 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 28121
I20260807 11:29:39.801856 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 28277
W20260807 11:29:39.805485 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 28277: Not authorized: Client connection negotiation failed: client connection to 127.24.103.67:41215: server requires authentication, but client does not have Kerberos credentials available
I20260807 11:29:39.805655 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 28277
W20260807 11:29:39.805778 28442 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:29:39.803201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.24.103.67:41215)
0807 11:29:39.803890 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:39.803897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:39.803927 (+ 30us) server_negotiation.cc:407] Connection header received
0807 11:29:39.803991 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:39.803997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:39.804265 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:39.804461 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:39.805400 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56384: BlockingRecv error: recv got EOF from 127.0.0.1:56384 (error 108)
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":352,"threads_started":1}
I20260807 11:29:39.825322 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 27854
I20260807 11:29:39.826768 27928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:39.937137 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 27854
2026-08-07T11:29:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3957 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:29:40 dist-test-slave-wl2h krb5kdc[28446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:29:40 dist-test-slave-wl2h krb5kdc[28446](info): set up 2 sockets
Aug 07 11:29:40 dist-test-slave-wl2h krb5kdc[28446](info): commencing operation
krb5kdc: starting...
W20260807 11:29:41.140887 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:29:41 dist-test-slave-wl2h krb5kdc[28446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:29:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:29:41.388016 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42111
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:41.722877 28461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:41.723457 28461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:41.723599 28461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:41.731267 28461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:41.731384 28461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:41.731431 28461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:41.731489 28461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:41.731539 28461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:29:41.741925 28461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:41.743345 28461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:41.744664 28461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:41.757032 28466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:41.757020 28467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:41.757009 28469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:41.757723 28461 server_base.cc:1034] running on GCE node
I20260807 11:29:41.758636 28461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:41.760231 28461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:41.761464 28461 hybrid_clock.cc:648] HybridClock initialized: now 1786102181761398 us; error 58 us; skew 500 ppm
I20260807 11:29:41.761893 28461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:41 dist-test-slave-wl2h krb5kdc[28446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102181, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:41.765642 28461 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:29:41.767295 28461 webserver.cc:460] Webserver started at http://127.24.103.126:37771/ using document root <none> and password file <none>
I20260807 11:29:41.768059 28461 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:41.768180 28461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:41.768538 28461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:41.771806 28461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "fdf273f2940e4d1aac7504c1731043d1"
format_stamp: "Formatted at 2026-08-07 11:29:41 on dist-test-slave-wl2h"
server_key: "7250f5562b2314a12928bd26749fee9a"
server_key_iv: "9e73d05380df37bffdfdd0baa48a7b39"
server_key_version: "encryptionkey@0"
I20260807 11:29:41.772675 28461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "fdf273f2940e4d1aac7504c1731043d1"
format_stamp: "Formatted at 2026-08-07 11:29:41 on dist-test-slave-wl2h"
server_key: "7250f5562b2314a12928bd26749fee9a"
server_key_iv: "9e73d05380df37bffdfdd0baa48a7b39"
server_key_version: "encryptionkey@0"
I20260807 11:29:41.778200 28461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:29:41.782415 28477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:41.784020 28461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:41.784279 28461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "fdf273f2940e4d1aac7504c1731043d1"
format_stamp: "Formatted at 2026-08-07 11:29:41 on dist-test-slave-wl2h"
server_key: "7250f5562b2314a12928bd26749fee9a"
server_key_iv: "9e73d05380df37bffdfdd0baa48a7b39"
server_key_version: "encryptionkey@0"
I20260807 11:29:41.784505 28461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:41.803869 28461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:41.808689 28461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:41.809018 28461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:41.829375 28461 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:42111
I20260807 11:29:41.829409 28532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:42111 every 8 connection(s)
I20260807 11:29:41.830873 28461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:41.832917 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 28461
I20260807 11:29:41.833176 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:41.834034 24989 external_mini_cluster.cc:1428] Setting key 587adf7c01093e8b0302970c5eb5c4b0
I20260807 11:29:41.836064 28533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:29:41 dist-test-slave-wl2h krb5kdc[28446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:41.851218 28533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1: Bootstrap starting.
I20260807 11:29:41.854933 28533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:41.856084 28533 log.cc:824] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:41.859263 28533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1: No bootstrap required, opened a new log
I20260807 11:29:41.859498 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:41.836007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.24.103.126:42111)
0807 11:29:41.836949 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:41.836961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:41.837002 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:29:41.838656 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:41.838677 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:41.839141 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:41.839493 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:41.841489 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:41.842893 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:41.844753 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:41.845445 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:41.845595 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:41.850748 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:41.850757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:41.850772 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:41.850813 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:41.854398 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:41.855462 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:41.855470 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:41.855477 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:41.855678 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:41.856155 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:41.856160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:41.856162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:41.856965 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:41.857346 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:41.857841 (+ 495us) server_negotiation.cc:299] Negotiation successful
0807 11:29:41.858302 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":461,"threads_started":1}
I20260807 11:29:41.863770 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:29:41.867419 28533 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf273f2940e4d1aac7504c1731043d1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42111 } }
I20260807 11:29:41.867851 28533 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:41.867967 28533 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf273f2940e4d1aac7504c1731043d1, State: Initialized, Role: FOLLOWER
I20260807 11:29:41.868557 28533 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf273f2940e4d1aac7504c1731043d1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42111 } }
I20260807 11:29:41.868836 28533 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:41.868990 28533 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:41.869184 28533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:41.871547 28533 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf273f2940e4d1aac7504c1731043d1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42111 } }
I20260807 11:29:41.872120 28533 leader_election.cc:302] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdf273f2940e4d1aac7504c1731043d1; no voters:
I20260807 11:29:41.875428 28533 leader_election.cc:288] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:41.875507 28539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:41.877195 28539 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [term 1 LEADER]: Becoming Leader. State: Replica: fdf273f2940e4d1aac7504c1731043d1, State: Running, Role: LEADER
I20260807 11:29:41.877827 28539 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf273f2940e4d1aac7504c1731043d1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42111 } }
I20260807 11:29:41.878088 28533 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:41.880903 28541 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf273f2940e4d1aac7504c1731043d1. Latest consensus state: current_term: 1 leader_uuid: "fdf273f2940e4d1aac7504c1731043d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf273f2940e4d1aac7504c1731043d1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42111 } } }
I20260807 11:29:41.881326 28541 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:41.881968 28540 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdf273f2940e4d1aac7504c1731043d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf273f2940e4d1aac7504c1731043d1" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42111 } } }
I20260807 11:29:41.882270 28540 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1 [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:41.889129 28547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:41.894845 28547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:41.919153 28547 catalog_manager.cc:1347] Generated new cluster ID: b126dcb440a44393b98765e9e706db74
I20260807 11:29:41.919305 28547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:41.934737 28547 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:41.936884 28547 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:41.959419 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:41.951234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.24.103.126:42111)
0807 11:29:41.951458 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:41.951465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:41.951787 (+ 322us) server_negotiation.cc:407] Connection header received
0807 11:29:41.952044 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:41.952049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:41.952118 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:41.952248 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:41.953401 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:41.954261 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:41.955749 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:41.956188 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:41.956364 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:41.957647 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:41.957652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:41.957655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:41.958452 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:41.958509 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:41.958532 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:41.958659 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:41.958817 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 11:29:41.958932 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":77}
I20260807 11:29:41.959372 28547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdf273f2940e4d1aac7504c1731043d1: Generated new TSK 0
I20260807 11:29:41.960510 28547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:29:41.969640 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 28461
I20260807 11:29:41.971182 28528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:42.067234 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 28461
2026-08-07T11:29:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2125 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:29:42 dist-test-slave-wl2h krb5kdc[28572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:29:42 dist-test-slave-wl2h krb5kdc[28572](info): set up 2 sockets
Aug 07 11:29:42 dist-test-slave-wl2h krb5kdc[28572](info): commencing operation
krb5kdc: starting...
W20260807 11:29:43.266444 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:29:43 dist-test-slave-wl2h krb5kdc[28572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:29:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:29:43.517011 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35421
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:35421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:43.864858 28587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:43.865164 28587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:43.865342 28587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:43.872937 28587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:43.873062 28587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:43.873150 28587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:43.873274 28587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:43.873363 28587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:29:43.883620 28587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:35421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:35421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:43.885058 28587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:43.886451 28587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:43.899045 28593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:43.898980 28596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:43.898995 28594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:43.900169 28587 server_base.cc:1034] running on GCE node
I20260807 11:29:43.900964 28587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:43.902520 28587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:43.903752 28587 hybrid_clock.cc:648] HybridClock initialized: now 1786102183903709 us; error 31 us; skew 500 ppm
I20260807 11:29:43.904090 28587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:43 dist-test-slave-wl2h krb5kdc[28572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102183, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:43.911859 28587 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:29:43.913704 28587 webserver.cc:460] Webserver started at http://127.24.103.126:37685/ using document root <none> and password file <none>
I20260807 11:29:43.914502 28587 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:43.914669 28587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:43.915108 28587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:43.919193 28587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "0a27f718f9de473b8eb0d9a023c61550"
format_stamp: "Formatted at 2026-08-07 11:29:43 on dist-test-slave-wl2h"
server_key: "5fd1fa1d1aed2f3c449c536257209a25"
server_key_iv: "e974a112c031352a5828390a929a1910"
server_key_version: "encryptionkey@0"
I20260807 11:29:43.920118 28587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "0a27f718f9de473b8eb0d9a023c61550"
format_stamp: "Formatted at 2026-08-07 11:29:43 on dist-test-slave-wl2h"
server_key: "5fd1fa1d1aed2f3c449c536257209a25"
server_key_iv: "e974a112c031352a5828390a929a1910"
server_key_version: "encryptionkey@0"
I20260807 11:29:43.926385 28587 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 11:29:43.930248 28604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:43.931986 28587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 11:29:43.932216 28587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "0a27f718f9de473b8eb0d9a023c61550"
format_stamp: "Formatted at 2026-08-07 11:29:43 on dist-test-slave-wl2h"
server_key: "5fd1fa1d1aed2f3c449c536257209a25"
server_key_iv: "e974a112c031352a5828390a929a1910"
server_key_version: "encryptionkey@0"
I20260807 11:29:43.932420 28587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:43.954275 28587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:43.958843 28587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:43.959208 28587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:43.979650 28587 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:35421
I20260807 11:29:43.979709 28663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:35421 every 8 connection(s)
I20260807 11:29:43.981144 28587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:43.982555 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 28587
I20260807 11:29:43.982801 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:43.983697 24989 external_mini_cluster.cc:1428] Setting key 75fbd03730c705166eb679487d0ab00f
I20260807 11:29:43.986193 28664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:29:43 dist-test-slave-wl2h krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:43.999344 28664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550: Bootstrap starting.
I20260807 11:29:44.003427 28664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:44.004871 28664 log.cc:824] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:44.007952 28664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550: No bootstrap required, opened a new log
I20260807 11:29:44.009166 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:43.985863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.24.103.126:35421)
0807 11:29:43.986701 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:43.986715 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:43.986766 (+ 51us) server_negotiation.cc:407] Connection header received
0807 11:29:43.989058 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:43.989084 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:43.989572 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:43.990018 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:43.991426 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:43.993508 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:43.994792 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:43.995839 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:43.996040 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:44.000462 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:44.000473 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:44.000490 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:44.000534 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:44.004872 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:44.005737 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:44.005746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:44.005754 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:44.005939 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:44.006383 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:44.006388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:44.006391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:44.006924 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:44.007295 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:44.007736 (+ 441us) server_negotiation.cc:299] Negotiation successful
0807 11:29:44.008167 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":390,"threads_started":1}
I20260807 11:29:44.016719 28664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a27f718f9de473b8eb0d9a023c61550" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35421 } }
I20260807 11:29:44.017118 28664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:44.017238 28664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a27f718f9de473b8eb0d9a023c61550, State: Initialized, Role: FOLLOWER
I20260807 11:29:44.017800 28664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a27f718f9de473b8eb0d9a023c61550" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35421 } }
I20260807 11:29:44.018050 28664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:44.018201 28664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:44.018384 28664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:44.020864 28664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a27f718f9de473b8eb0d9a023c61550" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35421 } }
I20260807 11:29:44.021371 28664 leader_election.cc:302] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a27f718f9de473b8eb0d9a023c61550; no voters:
I20260807 11:29:44.021993 28664 leader_election.cc:288] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:44.022164 28669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:44.022908 28669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [term 1 LEADER]: Becoming Leader. State: Replica: 0a27f718f9de473b8eb0d9a023c61550, State: Running, Role: LEADER
I20260807 11:29:44.023545 28669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a27f718f9de473b8eb0d9a023c61550" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35421 } }
I20260807 11:29:44.024204 28664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:44.027432 28670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a27f718f9de473b8eb0d9a023c61550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a27f718f9de473b8eb0d9a023c61550" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35421 } } }
I20260807 11:29:44.027902 28670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:44.028678 28671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a27f718f9de473b8eb0d9a023c61550. Latest consensus state: current_term: 1 leader_uuid: "0a27f718f9de473b8eb0d9a023c61550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a27f718f9de473b8eb0d9a023c61550" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 35421 } } }
I20260807 11:29:44.029132 28671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550 [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:44.032029 28676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:44.037801 28676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:44.046494 28676 catalog_manager.cc:1347] Generated new cluster ID: fc3d4d864404416c8615a571718a5878
I20260807 11:29:44.046658 28676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:44.062992 28676 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:44.064913 28676 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:44.075274 28676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a27f718f9de473b8eb0d9a023c61550: Generated new TSK 0
I20260807 11:29:44.076304 28676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:29:44.134967 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35421
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:44.487619 28692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:44.487931 28692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:44.488029 28692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:44.495814 28692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:44.495940 28692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:44.496057 28692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:29:44.506263 28692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:44.507782 28692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:44.509550 28692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:44.524103 28699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:44.524050 28700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:44.524099 28703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:44.524786 28692 server_base.cc:1034] running on GCE node
I20260807 11:29:44.525585 28692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:44.526600 28692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:44.527801 28692 hybrid_clock.cc:648] HybridClock initialized: now 1786102184527746 us; error 42 us; skew 500 ppm
I20260807 11:29:44.528165 28692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:44 dist-test-slave-wl2h krb5kdc[28572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102184, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:44.531780 28692 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:29:44.533493 28692 webserver.cc:460] Webserver started at http://127.24.103.65:45579/ using document root <none> and password file <none>
I20260807 11:29:44.534312 28692 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:44.534471 28692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:44.534816 28692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:44.538213 28692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "e8e218d3b4224b26940a5ffe9f4ac16b"
format_stamp: "Formatted at 2026-08-07 11:29:44 on dist-test-slave-wl2h"
server_key: "4e293f9a56cbfc3ea2ea7f9a5b65f6cc"
server_key_iv: "6f32d7ce8ee6a41d894a4046e766974d"
server_key_version: "encryptionkey@0"
I20260807 11:29:44.539191 28692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "e8e218d3b4224b26940a5ffe9f4ac16b"
format_stamp: "Formatted at 2026-08-07 11:29:44 on dist-test-slave-wl2h"
server_key: "4e293f9a56cbfc3ea2ea7f9a5b65f6cc"
server_key_iv: "6f32d7ce8ee6a41d894a4046e766974d"
server_key_version: "encryptionkey@0"
I20260807 11:29:44.545331 28692 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:29:44.549623 28711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:44.551146 28692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:44.551319 28692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "e8e218d3b4224b26940a5ffe9f4ac16b"
format_stamp: "Formatted at 2026-08-07 11:29:44 on dist-test-slave-wl2h"
server_key: "4e293f9a56cbfc3ea2ea7f9a5b65f6cc"
server_key_iv: "6f32d7ce8ee6a41d894a4046e766974d"
server_key_version: "encryptionkey@0"
I20260807 11:29:44.551532 28692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:44.569407 28692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:44.573601 28692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:44.573922 28692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:44.574988 28692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:44.576447 28692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:44.576553 28692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:44.576728 28692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:44.576805 28692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:44.611974 28692 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:44945
I20260807 11:29:44.611987 28841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:44945 every 8 connection(s)
I20260807 11:29:44.613319 28692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:29:44.619711 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 28692
I20260807 11:29:44.619942 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:29:44.620632 24989 external_mini_cluster.cc:1428] Setting key 640315b07ce1d61488c055b0714fdce6
Aug 07 11:29:44 dist-test-slave-wl2h krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102184, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:44.634900 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:44.615961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:56935 (local address 127.24.103.126:35421)
0807 11:29:44.616431 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:44.616435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:44.617195 (+ 760us) server_negotiation.cc:407] Connection header received
0807 11:29:44.619007 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:44.619041 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:44.619143 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:44.619273 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:44.620905 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:44.621784 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:44.623412 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:44.623866 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:44.624018 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:44.628000 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:44.628005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:44.628009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:44.628047 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:44.631218 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:44.632116 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:44.632129 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:44.632132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:44.632277 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:44.632775 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:44.632781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:44.632784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:44.633167 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:44.633351 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:44.634213 (+ 862us) server_negotiation.cc:299] Negotiation successful
0807 11:29:44.634482 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":271,"threads_started":1}
I20260807 11:29:44.636104 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:44.616333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35421 (local address 127.24.103.65:56935)
0807 11:29:44.617030 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:44.617075 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:44.618743 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:44.619509 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:44.619517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:44.620142 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:44.620694 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:44.620711 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:44.621966 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:44.621978 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:44.623258 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:44.623271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:44.624113 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:44.624123 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:44.624376 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:44.625081 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:44.625114 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:44.627732 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:44.631387 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:44.631397 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:44.631415 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:44.631916 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:44.632404 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:44.632410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:44.632414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:44.632643 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:44.633323 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:44.633332 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:44.633746 (+ 414us) client_negotiation.cc:769] Sending connection context
0807 11:29:44.634104 (+ 358us) client_negotiation.cc:240] Negotiation successful
0807 11:29:44.634754 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":274,"threads_started":1}
I20260807 11:29:44.637949 28842 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35421
I20260807 11:29:44.638299 28842 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:44.639195 28842 heartbeater.cc:511] Master 127.24.103.126:35421 requested a full tablet report, sending...
I20260807 11:29:44.641737 28623 ts_manager.cc:194] Registered new tserver with Master: e8e218d3b4224b26940a5ffe9f4ac16b (127.24.103.65:44945)
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260807 11:29:44.644021 28623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:56935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:29:44.674911 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35421
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:45.021014 28848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:45.021345 28848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:45.021448 28848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:45.028851 28848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:45.028967 28848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:45.029088 28848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:29:45.039708 28848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:45.041043 28848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:45.042327 28848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:45.055774 28854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:45.055688 28856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:45.055750 28858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:45.056577 28848 server_base.cc:1034] running on GCE node
I20260807 11:29:45.057181 28848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:45.058251 28848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:45.059459 28848 hybrid_clock.cc:648] HybridClock initialized: now 1786102185059415 us; error 41 us; skew 500 ppm
I20260807 11:29:45.059788 28848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:45 dist-test-slave-wl2h krb5kdc[28572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102185, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:45.063160 28848 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:29:45.064589 28848 webserver.cc:460] Webserver started at http://127.24.103.66:33367/ using document root <none> and password file <none>
I20260807 11:29:45.065282 28848 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:45.065415 28848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:45.065754 28848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:45.068742 28848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "69d0cbceff554d4dbc5e6394597219d7"
format_stamp: "Formatted at 2026-08-07 11:29:45 on dist-test-slave-wl2h"
server_key: "0127afac58e73ccb222b448208de979a"
server_key_iv: "ea908ebf02dd775ebb1206364b8e21be"
server_key_version: "encryptionkey@0"
I20260807 11:29:45.069546 28848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "69d0cbceff554d4dbc5e6394597219d7"
format_stamp: "Formatted at 2026-08-07 11:29:45 on dist-test-slave-wl2h"
server_key: "0127afac58e73ccb222b448208de979a"
server_key_iv: "ea908ebf02dd775ebb1206364b8e21be"
server_key_version: "encryptionkey@0"
I20260807 11:29:45.074824 28848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:29:45.078200 28865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:45.079612 28848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:45.079828 28848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "69d0cbceff554d4dbc5e6394597219d7"
format_stamp: "Formatted at 2026-08-07 11:29:45 on dist-test-slave-wl2h"
server_key: "0127afac58e73ccb222b448208de979a"
server_key_iv: "ea908ebf02dd775ebb1206364b8e21be"
server_key_version: "encryptionkey@0"
I20260807 11:29:45.080001 28848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:45.109818 28848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:45.114545 28848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:45.114884 28848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:45.115999 28848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:45.117336 28848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:45.117410 28848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:45.117493 28848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:45.117545 28848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:45.153646 28848 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:36867
I20260807 11:29:45.153702 28995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:36867 every 8 connection(s)
I20260807 11:29:45.154973 28848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:29:45.159622 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 28848
I20260807 11:29:45.159825 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:29:45.160300 24989 external_mini_cluster.cc:1428] Setting key 2b0d858672cd16e108016ea822f4bdb0
Aug 07 11:29:45 dist-test-slave-wl2h krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102185, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:45.177242 28998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:45.157696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:35023 (local address 127.24.103.126:35421)
0807 11:29:45.158137 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:45.158143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:45.159145 (+ 1002us) server_negotiation.cc:407] Connection header received
0807 11:29:45.161135 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:45.161141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:45.161247 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:45.161386 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:45.163039 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.164263 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.165898 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.166668 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.166817 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:45.170473 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:45.170482 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:45.170486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:45.170531 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:45.173336 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.174354 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.174358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.174360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.174495 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.175156 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.175160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.175162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.175494 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:45.175625 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:45.176669 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0807 11:29:45.176860 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":220,"threads_started":1}
I20260807 11:29:45.178779 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:45.158241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35421 (local address 127.24.103.66:35023)
0807 11:29:45.158930 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:45.158968 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:45.160855 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:45.161615 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:45.161623 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:45.162253 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:45.162828 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:45.162846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.164470 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.164474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:45.165738 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:45.165750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.166897 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.166902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:45.167211 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:45.167957 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:45.167980 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:45.170216 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:45.173500 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:45.173512 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:45.173534 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:45.174214 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:45.174612 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:45.174620 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:45.174626 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:45.174901 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:45.175638 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:45.175649 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:45.176160 (+ 511us) client_negotiation.cc:769] Sending connection context
0807 11:29:45.176545 (+ 385us) client_negotiation.cc:240] Negotiation successful
0807 11:29:45.177179 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":274,"threads_started":1}
I20260807 11:29:45.180492 28996 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35421
I20260807 11:29:45.180827 28996 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:45.181598 28996 heartbeater.cc:511] Master 127.24.103.126:35421 requested a full tablet report, sending...
I20260807 11:29:45.183385 28623 ts_manager.cc:194] Registered new tserver with Master: 69d0cbceff554d4dbc5e6394597219d7 (127.24.103.66:36867)
I20260807 11:29:45.184314 28623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:35023
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:29:45.218149 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35421
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:45.562083 29003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:45.562424 29003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:45.562525 29003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:45.569914 29003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:45.570039 29003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:45.570153 29003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:29:45.580464 29003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:35421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:45.581826 29003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:45.583207 29003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:45.596496 29009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:45.596539 29012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:45.596589 29010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:45.597337 29003 server_base.cc:1034] running on GCE node
I20260807 11:29:45.597931 29003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:45.598964 29003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:45.600229 29003 hybrid_clock.cc:648] HybridClock initialized: now 1786102185600183 us; error 47 us; skew 500 ppm
I20260807 11:29:45.600544 29003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:45 dist-test-slave-wl2h krb5kdc[28572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102185, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:45.604079 29003 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:29:45.605600 29003 webserver.cc:460] Webserver started at http://127.24.103.67:39773/ using document root <none> and password file <none>
I20260807 11:29:45.606324 29003 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:45.606480 29003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:45.606818 29003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:45.609961 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "1545d74ad0864fab87bf6d1d45c12627"
format_stamp: "Formatted at 2026-08-07 11:29:45 on dist-test-slave-wl2h"
server_key: "76b4874cbf845ca3f250cc9188941cde"
server_key_iv: "1f013ae6eb9322186d1307372e4ec41f"
server_key_version: "encryptionkey@0"
I20260807 11:29:45.610823 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "1545d74ad0864fab87bf6d1d45c12627"
format_stamp: "Formatted at 2026-08-07 11:29:45 on dist-test-slave-wl2h"
server_key: "76b4874cbf845ca3f250cc9188941cde"
server_key_iv: "1f013ae6eb9322186d1307372e4ec41f"
server_key_version: "encryptionkey@0"
I20260807 11:29:45.616608 29003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 11:29:45.620285 29019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:45.621728 29003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 11:29:45.621949 29003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "1545d74ad0864fab87bf6d1d45c12627"
format_stamp: "Formatted at 2026-08-07 11:29:45 on dist-test-slave-wl2h"
server_key: "76b4874cbf845ca3f250cc9188941cde"
server_key_iv: "1f013ae6eb9322186d1307372e4ec41f"
server_key_version: "encryptionkey@0"
I20260807 11:29:45.622175 29003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:45.643535 29003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:45.647274 28842 heartbeater.cc:503] Master 127.24.103.126:35421 was elected leader, sending a full tablet report...
I20260807 11:29:45.648247 29003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:45.648590 29003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:45.649814 29003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:45.651305 29003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:45.651384 29003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:45.651485 29003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:45.651530 29003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:45.686551 29003 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:33003
I20260807 11:29:45.686604 29153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:33003 every 8 connection(s)
I20260807 11:29:45.687985 29003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:29:45.693033 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 29003
I20260807 11:29:45.693256 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:29:45.693747 24989 external_mini_cluster.cc:1428] Setting key 5c9ead6695ae7689d87ae6bba2be36f4
Aug 07 11:29:45 dist-test-slave-wl2h krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102185, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:45.708950 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:45.690822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:41325 (local address 127.24.103.126:35421)
0807 11:29:45.691339 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:45.691344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:45.692294 (+ 950us) server_negotiation.cc:407] Connection header received
0807 11:29:45.694287 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:45.694293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:45.694406 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:45.694533 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:45.696213 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.697151 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.698684 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.699183 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.699293 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:45.702841 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:45.702846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:45.702849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:45.702889 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:45.705512 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.706230 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.706234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.706236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.706378 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.706825 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.706828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.706830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.707398 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:45.707566 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:45.708311 (+ 745us) server_negotiation.cc:299] Negotiation successful
0807 11:29:45.708542 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":287,"threads_started":1}
I20260807 11:29:45.710021 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:45.691392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:35421 (local address 127.24.103.67:41325)
0807 11:29:45.692120 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:45.692168 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:45.694015 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:45.694778 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:45.694786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:45.695406 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:45.696016 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:45.696035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.697298 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.697302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:45.698539 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:45.698552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.699418 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.699424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:45.699641 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:45.700446 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:45.700473 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:45.702642 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:45.705650 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:45.705657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:45.705670 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:45.706105 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:45.706507 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:45.706511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:45.706513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:45.706719 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:45.707543 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:45.707552 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:45.707929 (+ 377us) client_negotiation.cc:769] Sending connection context
0807 11:29:45.708217 (+ 288us) client_negotiation.cc:240] Negotiation successful
0807 11:29:45.708774 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":275,"threads_started":1}
I20260807 11:29:45.711469 29154 heartbeater.cc:347] Connected to a master server at 127.24.103.126:35421
I20260807 11:29:45.711792 29154 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:45.712651 29154 heartbeater.cc:511] Master 127.24.103.126:35421 requested a full tablet report, sending...
I20260807 11:29:45.714159 28623 ts_manager.cc:194] Registered new tserver with Master: 1545d74ad0864fab87bf6d1d45c12627 (127.24.103.67:33003)
I20260807 11:29:45.715271 28623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:41325
I20260807 11:29:45.721940 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:29:45 dist-test-slave-wl2h krb5kdc[28572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102185, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 11:29:45 dist-test-slave-wl2h krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102185, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:45.780938 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:45.764742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46522 (local address 127.24.103.126:35421)
0807 11:29:45.764953 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:45.764957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:45.765678 (+ 721us) server_negotiation.cc:407] Connection header received
0807 11:29:45.765843 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:45.765848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:45.765933 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:45.766034 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:45.767853 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.768755 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.770262 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.770715 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.770831 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:45.775245 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:45.775250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:45.775253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:45.775292 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:45.777991 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.778724 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.778729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.778732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.778894 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.779469 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.779473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.779475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.779876 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:45.780158 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:45.780516 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 11:29:45.780706 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260807 11:29:45.781698 28623 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46522
I20260807 11:29:45.783427 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 28692
Aug 07 11:29:45 dist-test-slave-wl2h krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102185, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:29:45.801363 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:45.784723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.24.103.65:44945)
0807 11:29:45.785472 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:45.785479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:45.785505 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:29:45.785564 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:45.785571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:45.785794 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:45.785921 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:45.787141 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.788174 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.789667 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.790412 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.790546 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:45.795257 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:45.795272 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:45.795287 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:45.795329 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:45.798369 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.799273 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.799281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.799284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.799433 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.799816 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.799819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.799821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.800195 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:45.800440 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:45.800698 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 11:29:45.800912 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":433,"threads_started":1}
W20260807 11:29:45.802874 28836 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45808
W20260807 11:29:45.804117 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 28692: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:29:45.804328 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 28692
I20260807 11:29:45.824312 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 28848
Aug 07 11:29:45 dist-test-slave-wl2h krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102185, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:29:45.842856 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:45.825328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.24.103.66:36867)
0807 11:29:45.825928 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:45.825935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:45.825962 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:29:45.826023 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:45.826029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:45.826247 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:45.826381 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:45.828721 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.829747 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.831247 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.831805 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.831909 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:45.836629 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:45.836643 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:45.836658 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:45.836699 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:45.839846 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.840571 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.840578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.840580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.840715 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.841131 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.841135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.841137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.841501 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:45.841847 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:45.842157 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 11:29:45.842441 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":318,"threads_started":1}
W20260807 11:29:45.844128 28990 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40384
W20260807 11:29:45.845202 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 28848: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:29:45.845345 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 28848
I20260807 11:29:45.864675 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 29003
Aug 07 11:29:45 dist-test-slave-wl2h krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102185, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:29:45.882225 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:45.865653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.24.103.67:33003)
0807 11:29:45.866341 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:45.866348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:45.866379 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:29:45.866441 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:45.866448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:45.866667 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:45.866814 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:45.868275 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.869351 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.870703 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:45.871480 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:45.871613 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:45.876305 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:45.876317 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:45.876333 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:45.876372 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:45.879570 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.880251 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.880259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.880261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.880386 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:45.880758 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:45.880761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:45.880763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:45.881104 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:45.881383 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:45.881691 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 11:29:45.881898 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":353,"threads_started":1}
W20260807 11:29:45.883562 29148 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58406
W20260807 11:29:45.884711 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 29003: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:29:45.884871 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 29003
I20260807 11:29:45.904371 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 28587
I20260807 11:29:45.905743 28658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:46.012979 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 28587
2026-08-07T11:29:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3946 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:29:46 dist-test-slave-wl2h krb5kdc[29175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:29:46 dist-test-slave-wl2h krb5kdc[29175](info): set up 2 sockets
Aug 07 11:29:46 dist-test-slave-wl2h krb5kdc[29175](info): commencing operation
krb5kdc: starting...
W20260807 11:29:47.196866 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:29:47 dist-test-slave-wl2h krb5kdc[29175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:29:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:29:47.455308 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:33557
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:47.796442 29189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:47.796757 29189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:47.796924 29189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:47.804536 29189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:47.804667 29189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:47.804754 29189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:47.804862 29189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:47.804940 29189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:29:47.815909 29189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:47.817430 29189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:47.818816 29189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:47.832573 29199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:47.832866 29196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:47.833055 29197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:47.834128 29189 server_base.cc:1034] running on GCE node
I20260807 11:29:47.835158 29189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:47.836825 29189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:47.838120 29189 hybrid_clock.cc:648] HybridClock initialized: now 1786102187838056 us; error 72 us; skew 500 ppm
I20260807 11:29:47.838524 29189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:47 dist-test-slave-wl2h krb5kdc[29175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102187, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:47.842372 29189 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:29:47.844446 29189 webserver.cc:460] Webserver started at http://127.24.103.126:40351/ using document root <none> and password file <none>
I20260807 11:29:47.845171 29189 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:47.845283 29189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:47.845554 29189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:47.848800 29189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "a87d08c5ab334fa38629978816c7c2eb"
format_stamp: "Formatted at 2026-08-07 11:29:47 on dist-test-slave-wl2h"
server_key: "6158351cbabf65935505b03faeb28705"
server_key_iv: "53c260d7143a00c5a90f35c03f560f01"
server_key_version: "encryptionkey@0"
I20260807 11:29:47.849558 29189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "a87d08c5ab334fa38629978816c7c2eb"
format_stamp: "Formatted at 2026-08-07 11:29:47 on dist-test-slave-wl2h"
server_key: "6158351cbabf65935505b03faeb28705"
server_key_iv: "53c260d7143a00c5a90f35c03f560f01"
server_key_version: "encryptionkey@0"
I20260807 11:29:47.855394 29189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 11:29:47.859366 29208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:47.861121 29189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:29:47.861327 29189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "a87d08c5ab334fa38629978816c7c2eb"
format_stamp: "Formatted at 2026-08-07 11:29:47 on dist-test-slave-wl2h"
server_key: "6158351cbabf65935505b03faeb28705"
server_key_iv: "53c260d7143a00c5a90f35c03f560f01"
server_key_version: "encryptionkey@0"
I20260807 11:29:47.861541 29189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:47.883780 29189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:47.888482 29189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:47.888819 29189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:47.908417 29189 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:33557
I20260807 11:29:47.908450 29264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:33557 every 8 connection(s)
I20260807 11:29:47.909811 29189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:47.911473 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 29189
I20260807 11:29:47.911720 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:47.912595 24989 external_mini_cluster.cc:1428] Setting key 4b721f3690954fb97f2f9a158498ad2f
I20260807 11:29:47.914656 29265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:29:47 dist-test-slave-wl2h krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:47.928745 29265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb: Bootstrap starting.
I20260807 11:29:47.932055 29265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:47.933327 29265 log.cc:824] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:47.936708 29265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb: No bootstrap required, opened a new log
I20260807 11:29:47.937220 29268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:47.914711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.24.103.126:33557)
0807 11:29:47.915565 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:47.915576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:47.915618 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:29:47.917207 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:47.917227 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:47.917595 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:47.917925 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:47.919886 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:47.921337 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:47.923169 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:47.923840 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:47.923965 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:47.929235 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:47.929243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:47.929256 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:47.929293 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:47.932480 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:47.933488 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:47.933500 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:47.933505 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:47.933694 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:47.934164 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:47.934171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:47.934175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:47.934954 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:47.935285 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:47.935726 (+ 441us) server_negotiation.cc:299] Negotiation successful
0807 11:29:47.936167 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":403,"threads_started":1}
I20260807 11:29:47.944276 29265 raft_consensus.cc:348] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87d08c5ab334fa38629978816c7c2eb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33557 } }
I20260807 11:29:47.944622 29265 raft_consensus.cc:374] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:47.944749 29265 raft_consensus.cc:729] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a87d08c5ab334fa38629978816c7c2eb, State: Initialized, Role: FOLLOWER
I20260807 11:29:47.945302 29265 consensus_queue.cc:260] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87d08c5ab334fa38629978816c7c2eb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33557 } }
I20260807 11:29:47.945500 29265 raft_consensus.cc:388] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:47.945652 29265 raft_consensus.cc:482] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:47.945833 29265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:47.948066 29265 raft_consensus.cc:504] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87d08c5ab334fa38629978816c7c2eb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33557 } }
I20260807 11:29:47.948555 29265 leader_election.cc:302] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a87d08c5ab334fa38629978816c7c2eb; no voters:
I20260807 11:29:47.949115 29265 leader_election.cc:288] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:47.949277 29271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:47.950058 29271 raft_consensus.cc:686] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [term 1 LEADER]: Becoming Leader. State: Replica: a87d08c5ab334fa38629978816c7c2eb, State: Running, Role: LEADER
I20260807 11:29:47.950645 29271 consensus_queue.cc:237] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87d08c5ab334fa38629978816c7c2eb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33557 } }
I20260807 11:29:47.951329 29265 sys_catalog.cc:564] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:47.953876 29272 sys_catalog.cc:455] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a87d08c5ab334fa38629978816c7c2eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87d08c5ab334fa38629978816c7c2eb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33557 } } }
I20260807 11:29:47.954365 29272 sys_catalog.cc:458] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:47.954780 29273 sys_catalog.cc:455] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a87d08c5ab334fa38629978816c7c2eb. Latest consensus state: current_term: 1 leader_uuid: "a87d08c5ab334fa38629978816c7c2eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87d08c5ab334fa38629978816c7c2eb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33557 } } }
I20260807 11:29:47.955096 29273 sys_catalog.cc:458] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:47.956885 29281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:47.961259 29281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:47.969767 29281 catalog_manager.cc:1347] Generated new cluster ID: 16bdbcf391b04815bc9999fc59a32b85
I20260807 11:29:47.969898 29281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:47.988047 29281 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:47.990123 29281 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:48.000455 29281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a87d08c5ab334fa38629978816c7c2eb: Generated new TSK 0
I20260807 11:29:48.001605 29281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:29:48.063426 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33557
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:48.407888 29294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:48.408215 29294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:48.408385 29294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:48.416018 29294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:48.416139 29294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:48.416356 29294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:29:48.426879 29294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:48.428285 29294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:48.429595 29294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:48.444199 29302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:48.444147 29300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:48.444332 29299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:48.445303 29294 server_base.cc:1034] running on GCE node
I20260807 11:29:48.446036 29294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:48.447242 29294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:48.448469 29294 hybrid_clock.cc:648] HybridClock initialized: now 1786102188448413 us; error 42 us; skew 500 ppm
I20260807 11:29:48.448868 29294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:48 dist-test-slave-wl2h krb5kdc[29175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102188, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:48.452945 29294 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:29:48.454510 29294 webserver.cc:460] Webserver started at http://127.24.103.65:42929/ using document root <none> and password file <none>
I20260807 11:29:48.455432 29294 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:48.455576 29294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:48.455989 29294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:48.459339 29294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "3860e56a7e5a47b6ab9ce61f68fe0eab"
format_stamp: "Formatted at 2026-08-07 11:29:48 on dist-test-slave-wl2h"
server_key: "64fefcf393361ff3d239b96c34e141de"
server_key_iv: "3f4575846f34d88c67e88b26ea33ea6f"
server_key_version: "encryptionkey@0"
I20260807 11:29:48.460299 29294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "3860e56a7e5a47b6ab9ce61f68fe0eab"
format_stamp: "Formatted at 2026-08-07 11:29:48 on dist-test-slave-wl2h"
server_key: "64fefcf393361ff3d239b96c34e141de"
server_key_iv: "3f4575846f34d88c67e88b26ea33ea6f"
server_key_version: "encryptionkey@0"
I20260807 11:29:48.466656 29294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 11:29:48.470381 29310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:48.472167 29294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:48.472399 29294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "3860e56a7e5a47b6ab9ce61f68fe0eab"
format_stamp: "Formatted at 2026-08-07 11:29:48 on dist-test-slave-wl2h"
server_key: "64fefcf393361ff3d239b96c34e141de"
server_key_iv: "3f4575846f34d88c67e88b26ea33ea6f"
server_key_version: "encryptionkey@0"
I20260807 11:29:48.472651 29294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:48.492117 29294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:48.497424 29294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:48.497782 29294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:48.498932 29294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:48.500837 29294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:48.500958 29294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:48.501130 29294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:48.501221 29294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:48.539505 29294 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:35593
I20260807 11:29:48.539568 29439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:35593 every 8 connection(s)
I20260807 11:29:48.540946 29294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:29:48.548950 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 29294
I20260807 11:29:48.549216 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:29:48.550125 24989 external_mini_cluster.cc:1428] Setting key 4ed4d6d9b91c35d9f81393461ecb6bf4
Aug 07 11:29:48 dist-test-slave-wl2h krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102188, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:48.565322 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:48.543860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:58069 (local address 127.24.103.126:33557)
0807 11:29:48.544415 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:48.544422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:48.545372 (+ 950us) server_negotiation.cc:407] Connection header received
0807 11:29:48.547685 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:48.547692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:48.547785 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:48.547911 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:48.550225 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:48.551448 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:48.553251 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:48.553995 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:48.554152 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:48.557995 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:48.558003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:48.558010 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:48.558055 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:48.562006 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:48.562810 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:48.562816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:48.562820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:48.562952 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:48.563438 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:48.563444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:48.563448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:48.563743 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:48.563903 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:48.564711 (+ 808us) server_negotiation.cc:299] Negotiation successful
0807 11:29:48.564942 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":279,"threads_started":1}
I20260807 11:29:48.566358 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:48.544327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33557 (local address 127.24.103.65:58069)
0807 11:29:48.545188 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:48.545234 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:48.547371 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:48.548209 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:48.548219 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:48.549037 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:48.549930 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:48.549961 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:48.551602 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:48.551608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:48.553088 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:48.553101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:48.554308 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:48.554314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:48.554599 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:48.555410 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:48.555434 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:48.557730 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:48.562168 (+ 4438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:48.562178 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:48.562197 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:48.562676 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:48.563098 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:48.563104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:48.563107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:48.563317 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:48.563893 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:48.563902 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:48.564277 (+ 375us) client_negotiation.cc:769] Sending connection context
0807 11:29:48.564605 (+ 328us) client_negotiation.cc:240] Negotiation successful
0807 11:29:48.565213 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":326,"threads_started":1}
I20260807 11:29:48.568142 29440 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33557
I20260807 11:29:48.568497 29440 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:48.569422 29440 heartbeater.cc:511] Master 127.24.103.126:33557 requested a full tablet report, sending...
I20260807 11:29:48.571988 29226 ts_manager.cc:194] Registered new tserver with Master: 3860e56a7e5a47b6ab9ce61f68fe0eab (127.24.103.65:35593)
I20260807 11:29:48.573450 29226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:58069
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:29:48.610957 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33557
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:48.948642 29450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:48.948961 29450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:48.949148 29450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:48.957149 29450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:48.957296 29450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:48.957479 29450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:29:48.967381 29450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:48.968696 29450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:48.970180 29450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:48.983852 29458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:48.983981 29456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:48.983942 29455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:48.984676 29450 server_base.cc:1034] running on GCE node
I20260807 11:29:48.985343 29450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:48.986397 29450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:48.987684 29450 hybrid_clock.cc:648] HybridClock initialized: now 1786102188987595 us; error 101 us; skew 500 ppm
I20260807 11:29:48.988040 29450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:48 dist-test-slave-wl2h krb5kdc[29175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102188, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:48.994879 29450 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:29:48.996491 29450 webserver.cc:460] Webserver started at http://127.24.103.66:42847/ using document root <none> and password file <none>
I20260807 11:29:48.997216 29450 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:48.997323 29450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:48.997680 29450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:49.000828 29450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "81e1df3754904d6fb396ec61edfd2d6e"
format_stamp: "Formatted at 2026-08-07 11:29:48 on dist-test-slave-wl2h"
server_key: "9cfd55cbbca28a3ef92acc63d42d4bf7"
server_key_iv: "bff89403a7f1a13b434dfde5524b2e0a"
server_key_version: "encryptionkey@0"
I20260807 11:29:49.001683 29450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "81e1df3754904d6fb396ec61edfd2d6e"
format_stamp: "Formatted at 2026-08-07 11:29:48 on dist-test-slave-wl2h"
server_key: "9cfd55cbbca28a3ef92acc63d42d4bf7"
server_key_iv: "bff89403a7f1a13b434dfde5524b2e0a"
server_key_version: "encryptionkey@0"
I20260807 11:29:49.006984 29450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:29:49.010740 29467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:49.012341 29450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:49.012550 29450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "81e1df3754904d6fb396ec61edfd2d6e"
format_stamp: "Formatted at 2026-08-07 11:29:48 on dist-test-slave-wl2h"
server_key: "9cfd55cbbca28a3ef92acc63d42d4bf7"
server_key_iv: "bff89403a7f1a13b434dfde5524b2e0a"
server_key_version: "encryptionkey@0"
I20260807 11:29:49.012755 29450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:49.047932 29450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:49.052644 29450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:49.052969 29450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:49.053987 29450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:49.055693 29450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:49.055886 29450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260807 11:29:49.056034 29450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:49.056085 29450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:49.091463 29450 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:34775
I20260807 11:29:49.091564 29593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:34775 every 8 connection(s)
I20260807 11:29:49.092811 29450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:29:49.096808 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 29450
I20260807 11:29:49.097015 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:29:49.097612 24989 external_mini_cluster.cc:1428] Setting key b6d77fe19688a014d300e649fe0761dd
Aug 07 11:29:49 dist-test-slave-wl2h krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102188, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:49.116547 29596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:49.095615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:55975 (local address 127.24.103.126:33557)
0807 11:29:49.096127 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:49.096132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:49.099110 (+ 2978us) server_negotiation.cc:407] Connection header received
0807 11:29:49.099345 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:49.099351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:49.099437 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:49.099555 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:49.101608 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.102332 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.104869 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.105389 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.105506 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:49.110030 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:49.110035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:49.110038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:49.110075 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:49.112847 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.113833 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.113837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.113839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.113962 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.114492 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.114495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.114497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.114763 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:49.114870 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:49.115910 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0807 11:29:49.116151 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":249,"threads_started":1}
I20260807 11:29:49.118068 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:49.096013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33557 (local address 127.24.103.66:55975)
0807 11:29:49.096959 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:49.097021 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:49.099000 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:49.099836 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:49.099848 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:49.100584 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:49.101403 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:49.101431 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.102502 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.102510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:49.104695 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:49.104714 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.105661 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.105688 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:49.106040 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:49.106867 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:49.106916 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:49.109708 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:49.113002 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:49.113014 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:49.113047 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:49.113700 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:49.114087 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:49.114095 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:49.114101 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:49.114377 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:49.114892 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:49.114903 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:49.115439 (+ 536us) client_negotiation.cc:769] Sending connection context
0807 11:29:49.115813 (+ 374us) client_negotiation.cc:240] Negotiation successful
0807 11:29:49.116462 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":404,"threads_started":1}
I20260807 11:29:49.119611 29594 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33557
I20260807 11:29:49.119982 29594 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:49.120954 29594 heartbeater.cc:511] Master 127.24.103.126:33557 requested a full tablet report, sending...
I20260807 11:29:49.122697 29226 ts_manager.cc:194] Registered new tserver with Master: 81e1df3754904d6fb396ec61edfd2d6e (127.24.103.66:34775)
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260807 11:29:49.123713 29226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:55975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:29:49.158540 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33557
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:49.500895 29601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:49.501247 29601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:49.501430 29601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:49.509483 29601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:49.509690 29601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:49.509913 29601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:29:49.520666 29601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:49.522064 29601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:49.523452 29601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:49.543604 29607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:49.543689 29610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:49.543638 29608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:49.544832 29601 server_base.cc:1034] running on GCE node
I20260807 11:29:49.545462 29601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:49.546504 29601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:49.547765 29601 hybrid_clock.cc:648] HybridClock initialized: now 1786102189547717 us; error 32 us; skew 500 ppm
I20260807 11:29:49.548153 29601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:49 dist-test-slave-wl2h krb5kdc[29175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102189, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:49.551739 29601 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:29:49.553490 29601 webserver.cc:460] Webserver started at http://127.24.103.67:46197/ using document root <none> and password file <none>
I20260807 11:29:49.554240 29601 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:49.554371 29601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:49.554749 29601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:49.557904 29601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "928582d9d4a1424eb5208af02acfccd2"
format_stamp: "Formatted at 2026-08-07 11:29:49 on dist-test-slave-wl2h"
server_key: "ad17a2c5379215f2a5dcde2eb2878a04"
server_key_iv: "1d84cde26159fc4c4d3c8166b4989089"
server_key_version: "encryptionkey@0"
I20260807 11:29:49.558707 29601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "928582d9d4a1424eb5208af02acfccd2"
format_stamp: "Formatted at 2026-08-07 11:29:49 on dist-test-slave-wl2h"
server_key: "ad17a2c5379215f2a5dcde2eb2878a04"
server_key_iv: "1d84cde26159fc4c4d3c8166b4989089"
server_key_version: "encryptionkey@0"
I20260807 11:29:49.564666 29601 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260807 11:29:49.568651 29618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:49.570065 29601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:29:49.570429 29601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "928582d9d4a1424eb5208af02acfccd2"
format_stamp: "Formatted at 2026-08-07 11:29:49 on dist-test-slave-wl2h"
server_key: "ad17a2c5379215f2a5dcde2eb2878a04"
server_key_iv: "1d84cde26159fc4c4d3c8166b4989089"
server_key_version: "encryptionkey@0"
I20260807 11:29:49.570691 29601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:49.577014 29440 heartbeater.cc:503] Master 127.24.103.126:33557 was elected leader, sending a full tablet report...
I20260807 11:29:49.585074 29601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:49.589088 29601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:49.589335 29601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:49.590232 29601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:49.591581 29601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:49.591656 29601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:49.591740 29601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:49.591789 29601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:49.626766 29601 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:35391
I20260807 11:29:49.626914 29750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:35391 every 8 connection(s)
I20260807 11:29:49.628206 29601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:29:49.634419 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 29601
I20260807 11:29:49.634663 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:29:49.635174 24989 external_mini_cluster.cc:1428] Setting key 873d88ef1db83fd88ff6f40498ada02e
Aug 07 11:29:49 dist-test-slave-wl2h krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102189, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:49.650266 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:49.631139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:44289 (local address 127.24.103.126:33557)
0807 11:29:49.631604 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:49.631610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:49.632587 (+ 977us) server_negotiation.cc:407] Connection header received
0807 11:29:49.634506 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:49.634512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:49.634648 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:49.634773 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:49.636815 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.637907 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.639897 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.640393 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.640494 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:49.644032 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:49.644037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:49.644044 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:49.644121 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:49.646655 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.647536 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.647542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.647545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.647689 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.648329 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.648334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.648336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.648617 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:49.648754 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:49.649573 (+ 819us) server_negotiation.cc:299] Negotiation successful
0807 11:29:49.649805 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":228,"threads_started":1}
I20260807 11:29:49.651319 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:49.631629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33557 (local address 127.24.103.67:44289)
0807 11:29:49.632404 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:49.632443 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:49.634199 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:49.635075 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:49.635095 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:49.635726 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:49.636483 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:49.636516 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.638083 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.638088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:49.639745 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:49.639758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.640606 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.640611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:49.640817 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:49.641501 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:49.641525 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:49.643797 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:49.646832 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:49.646842 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:49.646856 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:49.647380 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:49.647808 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:49.647812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:49.647814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:49.647997 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:49.648746 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:49.648752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:49.649093 (+ 341us) client_negotiation.cc:769] Sending connection context
0807 11:29:49.649474 (+ 381us) client_negotiation.cc:240] Negotiation successful
0807 11:29:49.650041 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":308,"threads_started":1}
I20260807 11:29:49.652733 29751 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33557
I20260807 11:29:49.653115 29751 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:49.653918 29751 heartbeater.cc:511] Master 127.24.103.126:33557 requested a full tablet report, sending...
I20260807 11:29:49.655793 29226 ts_manager.cc:194] Registered new tserver with Master: 928582d9d4a1424eb5208af02acfccd2 (127.24.103.67:35391)
I20260807 11:29:49.656750 29226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:44289
I20260807 11:29:49.664243 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:29:49 dist-test-slave-wl2h krb5kdc[29175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 11:29:49 dist-test-slave-wl2h krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:29:49.723399 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:49.707583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.24.103.65:35593)
0807 11:29:49.708276 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:49.708284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:49.708314 (+ 30us) server_negotiation.cc:407] Connection header received
0807 11:29:49.708378 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:49.708385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:49.708619 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:49.708752 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:49.710334 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.711527 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.712808 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.713385 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.713486 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:49.717732 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:49.717743 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:49.717758 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:49.717795 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:49.720726 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.721394 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.721401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.721404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.721546 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.721884 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.721887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.721888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.722239 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:49.722472 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:49.722731 (+ 259us) server_negotiation.cc:299] Negotiation successful
0807 11:29:49.722935 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":343,"threads_started":1}
Aug 07 11:29:49 dist-test-slave-wl2h krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:49.743266 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:49.728576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.24.103.126:33557)
0807 11:29:49.728799 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:49.728803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:49.728888 (+ 85us) server_negotiation.cc:407] Connection header received
0807 11:29:49.729032 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:49.729036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:49.729119 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:49.729214 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:49.730632 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.731517 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.732787 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.733239 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.733380 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:49.737647 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:49.737652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:49.737656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:49.737694 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:49.740443 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.741174 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.741178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.741181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.741315 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.741727 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.741732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.741737 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.742076 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:49.742329 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:49.742665 (+ 336us) server_negotiation.cc:299] Negotiation successful
0807 11:29:49.742904 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260807 11:29:49.743929 29226 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46044
I20260807 11:29:49.745000 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 29294
I20260807 11:29:49.758890 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:49.746254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.24.103.65:35593)
0807 11:29:49.746480 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:49.746487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:49.746589 (+ 102us) server_negotiation.cc:407] Connection header received
0807 11:29:49.746799 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:49.746803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:49.746868 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:49.746970 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:49.748257 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.748935 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.750166 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.750606 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.750761 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:49.753253 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:49.753258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:49.753261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:49.753295 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:49.756135 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.756797 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.756804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.756807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.756945 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.757455 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.757461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.757466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.757748 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:49.757930 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:49.758330 (+ 400us) server_negotiation.cc:299] Negotiation successful
0807 11:29:49.758502 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 11:29:49.759654 29435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:49.845005 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 29294
I20260807 11:29:49.867681 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 29450
Aug 07 11:29:49 dist-test-slave-wl2h krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:29:49.884828 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:49.868645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.24.103.66:34775)
0807 11:29:49.869223 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:49.869231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:49.869258 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:29:49.869320 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:49.869327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:49.869542 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:49.869678 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:49.870934 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.872106 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.873330 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:49.873948 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:49.874060 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:49.878594 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:49.878605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:49.878619 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:49.878654 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:49.881853 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.882557 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.882564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.882567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.882698 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:49.883169 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:49.883202 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:49.883205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:49.883643 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:49.883882 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:49.884258 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 11:29:49.884480 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":290,"threads_started":1}
I20260807 11:29:49.886322 29589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:49.977612 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 29450
I20260807 11:29:49.998639 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 29601
Aug 07 11:29:50 dist-test-slave-wl2h krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:29:50.017838 29766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:49.999858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49646 (local address 127.24.103.67:35391)
0807 11:29:50.000576 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:50.000583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:50.000615 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:29:50.000686 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:50.000692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:50.000926 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:50.001088 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:50.002492 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:50.003703 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:50.005178 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:50.005742 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:50.005842 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:50.010914 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:50.010928 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:50.010945 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:50.010986 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:50.014341 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:50.015543 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:50.015551 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:50.015554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:50.015718 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:50.016190 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:50.016195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:50.016197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:50.016582 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:50.016852 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:50.017304 (+ 452us) server_negotiation.cc:299] Negotiation successful
0807 11:29:50.017520 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":399,"threads_started":1}
I20260807 11:29:50.019342 29744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:50.104610 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 29601
I20260807 11:29:50.124579 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 29189
I20260807 11:29:50.125998 29260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:50.235121 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 29189
2026-08-07T11:29:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4226 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:29:50 dist-test-slave-wl2h krb5kdc[29772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:29:50 dist-test-slave-wl2h krb5kdc[29772](info): set up 2 sockets
Aug 07 11:29:50 dist-test-slave-wl2h krb5kdc[29772](info): commencing operation
krb5kdc: starting...
W20260807 11:29:51.427822 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:29:51 dist-test-slave-wl2h krb5kdc[29772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:29:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:29:51.679867 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42041
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:52.018384 29787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:52.018718 29787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:52.018814 29787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:52.026193 29787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:52.026299 29787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:52.026345 29787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:52.026400 29787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:52.026448 29787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:29:52.036566 29787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:52.038115 29787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:52.039456 29787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:52.051218 29793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:52.051411 29796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:52.051962 29794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:52.052390 29787 server_base.cc:1034] running on GCE node
I20260807 11:29:52.053108 29787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:52.054489 29787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:52.055691 29787 hybrid_clock.cc:648] HybridClock initialized: now 1786102192055639 us; error 46 us; skew 500 ppm
I20260807 11:29:52.056035 29787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:52 dist-test-slave-wl2h krb5kdc[29772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102192, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:52.059104 29787 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:29:52.060292 29787 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:52.060431 29787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:52.060765 29787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:52.063818 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "6f7cc67b460344b4955569d163794ad0"
format_stamp: "Formatted at 2026-08-07 11:29:52 on dist-test-slave-wl2h"
server_key: "c0e8637ab5f4690635533321e9661549"
server_key_iv: "4219569889d1cceec826cea601d0c943"
server_key_version: "encryptionkey@0"
I20260807 11:29:52.064698 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "6f7cc67b460344b4955569d163794ad0"
format_stamp: "Formatted at 2026-08-07 11:29:52 on dist-test-slave-wl2h"
server_key: "c0e8637ab5f4690635533321e9661549"
server_key_iv: "4219569889d1cceec826cea601d0c943"
server_key_version: "encryptionkey@0"
I20260807 11:29:52.069552 29787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:29:52.072928 29802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:52.074244 29787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:29:52.074556 29787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "6f7cc67b460344b4955569d163794ad0"
format_stamp: "Formatted at 2026-08-07 11:29:52 on dist-test-slave-wl2h"
server_key: "c0e8637ab5f4690635533321e9661549"
server_key_iv: "4219569889d1cceec826cea601d0c943"
server_key_version: "encryptionkey@0"
I20260807 11:29:52.074744 29787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:52.094035 29787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:52.098272 29787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:52.098502 29787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:52.118412 29787 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:42041
I20260807 11:29:52.118430 29867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:42041 every 8 connection(s)
I20260807 11:29:52.119414 29787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:52.123334 29868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:52.125829 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 29787
I20260807 11:29:52.126096 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:52.126792 24989 external_mini_cluster.cc:1428] Setting key eac249509fde432c1f79190bc34c3f63
I20260807 11:29:52.138160 29868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0: Bootstrap starting.
I20260807 11:29:52.142174 29868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:52.143352 29868 log.cc:824] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0: Log is configured to *not* fsync() on all Append() calls
Aug 07 11:29:52 dist-test-slave-wl2h krb5kdc[29772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:52.146626 29868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0: No bootstrap required, opened a new log
I20260807 11:29:52.152503 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:52.128896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.24.103.126:42041)
0807 11:29:52.129916 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:52.129932 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:52.129989 (+ 57us) server_negotiation.cc:407] Connection header received
0807 11:29:52.132647 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:52.132678 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:52.133248 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:52.133752 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:52.135355 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:52.137537 (+ 2182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:52.139010 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:52.140263 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:52.140489 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:52.144781 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:52.144790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:52.144805 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:52.144848 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:52.148235 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:52.148972 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:52.148979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:52.148985 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:52.149160 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:52.149578 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:52.149588 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:52.149590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:52.150182 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:52.150522 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:52.150991 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 11:29:52.151464 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":440,"threads_started":1}
I20260807 11:29:52.155071 29868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7cc67b460344b4955569d163794ad0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42041 } }
I20260807 11:29:52.155546 29868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:52.155668 29868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7cc67b460344b4955569d163794ad0, State: Initialized, Role: FOLLOWER
I20260807 11:29:52.156236 29868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7cc67b460344b4955569d163794ad0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42041 } }
I20260807 11:29:52.156436 29868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:52.156529 29868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:52.156652 29868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:52.159191 29868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7cc67b460344b4955569d163794ad0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42041 } }
I20260807 11:29:52.159778 29868 leader_election.cc:302] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f7cc67b460344b4955569d163794ad0; no voters:
I20260807 11:29:52.160578 29868 leader_election.cc:288] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:52.160813 29873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:52.163214 29868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:52.163686 29873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [term 1 LEADER]: Becoming Leader. State: Replica: 6f7cc67b460344b4955569d163794ad0, State: Running, Role: LEADER
I20260807 11:29:52.164337 29873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7cc67b460344b4955569d163794ad0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42041 } }
I20260807 11:29:52.167827 29874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f7cc67b460344b4955569d163794ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7cc67b460344b4955569d163794ad0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42041 } } }
I20260807 11:29:52.167670 29875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f7cc67b460344b4955569d163794ad0. Latest consensus state: current_term: 1 leader_uuid: "6f7cc67b460344b4955569d163794ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7cc67b460344b4955569d163794ad0" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42041 } } }
I20260807 11:29:52.168285 29875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:52.168406 29874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0 [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:52.169571 29878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:52.180847 29878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:52.197688 29878 catalog_manager.cc:1347] Generated new cluster ID: 05ac8b796a2b4f82b5210d5f4635012d
I20260807 11:29:52.197844 29878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:52.227627 29878 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:52.229194 29878 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:52.236644 29878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0: Generated new TSK 0
I20260807 11:29:52.237541 29878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:29:52.295836 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42041
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:52.648825 29896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:52.649121 29896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:52.649299 29896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:52.656956 29896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:52.657078 29896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:52.657289 29896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:29:52.667892 29896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:52.669323 29896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:52.670921 29896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:52.684723 29902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:52.684883 29906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:52.685405 29901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:52.685708 29896 server_base.cc:1034] running on GCE node
I20260807 11:29:52.686353 29896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:52.687460 29896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:52.688735 29896 hybrid_clock.cc:648] HybridClock initialized: now 1786102192688685 us; error 40 us; skew 500 ppm
I20260807 11:29:52.689086 29896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:52 dist-test-slave-wl2h krb5kdc[29772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102192, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:52.692529 29896 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:29:52.693797 29896 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:52.693936 29896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:52.694269 29896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:52.697346 29896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "3d0f7dd1518d418ea84f111fd8b32ae3"
format_stamp: "Formatted at 2026-08-07 11:29:52 on dist-test-slave-wl2h"
server_key: "6bcd2ca827f389d39608ac5e3650e199"
server_key_iv: "c27fd64586e35a6bc23c43a6c52d30da"
server_key_version: "encryptionkey@0"
I20260807 11:29:52.698184 29896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "3d0f7dd1518d418ea84f111fd8b32ae3"
format_stamp: "Formatted at 2026-08-07 11:29:52 on dist-test-slave-wl2h"
server_key: "6bcd2ca827f389d39608ac5e3650e199"
server_key_iv: "c27fd64586e35a6bc23c43a6c52d30da"
server_key_version: "encryptionkey@0"
I20260807 11:29:52.703567 29896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260807 11:29:52.707201 29912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:52.708601 29896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:29:52.708796 29896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "3d0f7dd1518d418ea84f111fd8b32ae3"
format_stamp: "Formatted at 2026-08-07 11:29:52 on dist-test-slave-wl2h"
server_key: "6bcd2ca827f389d39608ac5e3650e199"
server_key_iv: "c27fd64586e35a6bc23c43a6c52d30da"
server_key_version: "encryptionkey@0"
I20260807 11:29:52.709033 29896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:52.721786 29896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:52.725841 29896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:52.726143 29896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:52.727213 29896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:52.728591 29896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:52.728716 29896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:52.728859 29896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:52.728951 29896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:52.765493 29896 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:37017
I20260807 11:29:52.765537 30038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:37017 every 8 connection(s)
I20260807 11:29:52.766392 29896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:29:52.772514 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 29896
I20260807 11:29:52.772779 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:29:52.773679 24989 external_mini_cluster.cc:1428] Setting key 41e706820dd9a3f9bc2286741c7acbb3
Aug 07 11:29:52 dist-test-slave-wl2h krb5kdc[29772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102192, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:52.789201 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:52.769272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:60727 (local address 127.24.103.126:42041)
0807 11:29:52.769798 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:52.769806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:52.770710 (+ 904us) server_negotiation.cc:407] Connection header received
0807 11:29:52.772987 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:52.772991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:52.773064 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:52.773170 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:52.775191 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:52.776195 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:52.777842 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:52.778288 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:52.778427 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:52.782339 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:52.782344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:52.782347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:52.782380 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:52.785582 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:52.786404 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:52.786413 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:52.786419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:52.786590 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:52.787062 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:52.787071 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:52.787077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:52.787479 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:52.787649 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:52.788419 (+ 770us) server_negotiation.cc:299] Negotiation successful
0807 11:29:52.788701 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":374,"thread_start_us":231,"threads_started":1}
I20260807 11:29:52.789943 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:52.769689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42041 (local address 127.24.103.65:60727)
0807 11:29:52.770525 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:52.770577 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:52.772630 (+ 2053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:52.773470 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:52.773484 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:52.774273 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:52.774925 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:52.774950 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:52.776359 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:52.776372 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:52.777672 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:52.777685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:52.778560 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:52.778569 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:52.778784 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:52.779583 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:52.779613 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:52.782083 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:52.785770 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:52.785778 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:52.785793 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:52.786266 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:52.786717 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:52.786721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:52.786723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:52.786924 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:52.787656 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:52.787663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:52.788007 (+ 344us) client_negotiation.cc:769] Sending connection context
0807 11:29:52.788268 (+ 261us) client_negotiation.cc:240] Negotiation successful
0807 11:29:52.788685 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":361,"threads_started":1}
I20260807 11:29:52.791718 30039 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42041
I20260807 11:29:52.792079 30039 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:52.792797 30039 heartbeater.cc:511] Master 127.24.103.126:42041 requested a full tablet report, sending...
I20260807 11:29:52.795207 29824 ts_manager.cc:194] Registered new tserver with Master: 3d0f7dd1518d418ea84f111fd8b32ae3 (127.24.103.65:37017)
I20260807 11:29:52.796756 29824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:60727
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:29:52.835125 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42041
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:53.191043 30046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:53.191428 30046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:53.191618 30046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:53.199273 30046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:53.199440 30046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:53.199631 30046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:29:53.209712 30046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:53.211208 30046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:53.212677 30046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:53.226397 30054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:53.226490 30052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:53.226454 30051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:53.232865 30046 server_base.cc:1034] running on GCE node
I20260807 11:29:53.233575 30046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:53.234957 30046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:53.236384 30046 hybrid_clock.cc:648] HybridClock initialized: now 1786102193236315 us; error 46 us; skew 500 ppm
I20260807 11:29:53.236871 30046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:29:53 dist-test-slave-wl2h krb5kdc[29772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102193, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:53.240787 30046 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:29:53.242228 30046 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:53.242362 30046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:53.242753 30046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:53.246011 30046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "a8d89564cbeb4f98af3a8edf3490b757"
format_stamp: "Formatted at 2026-08-07 11:29:53 on dist-test-slave-wl2h"
server_key: "cb274757c6ff60fc178ca1f183f999d0"
server_key_iv: "9e34311ddefea154320be2c49357aac0"
server_key_version: "encryptionkey@0"
I20260807 11:29:53.246894 30046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "a8d89564cbeb4f98af3a8edf3490b757"
format_stamp: "Formatted at 2026-08-07 11:29:53 on dist-test-slave-wl2h"
server_key: "cb274757c6ff60fc178ca1f183f999d0"
server_key_iv: "9e34311ddefea154320be2c49357aac0"
server_key_version: "encryptionkey@0"
I20260807 11:29:53.252879 30046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260807 11:29:53.256636 30060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:53.258108 30046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:29:53.258354 30046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "a8d89564cbeb4f98af3a8edf3490b757"
format_stamp: "Formatted at 2026-08-07 11:29:53 on dist-test-slave-wl2h"
server_key: "cb274757c6ff60fc178ca1f183f999d0"
server_key_iv: "9e34311ddefea154320be2c49357aac0"
server_key_version: "encryptionkey@0"
I20260807 11:29:53.258606 30046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:53.290980 30046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:53.296798 30046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:53.297907 30046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:53.299170 30046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:53.300858 30046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:53.300988 30046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:53.301126 30046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:53.301215 30046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:53.339890 30046 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:46623
I20260807 11:29:53.339952 30183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:46623 every 8 connection(s)
I20260807 11:29:53.340811 30046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:29:53.343357 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 30046
I20260807 11:29:53.343590 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:29:53.344213 24989 external_mini_cluster.cc:1428] Setting key e10d6d7decd54ad63da68bdba9d3b3fa
Aug 07 11:29:53 dist-test-slave-wl2h krb5kdc[29772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102193, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:53.366839 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:53.344377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42041 (local address 127.24.103.66:47617)
0807 11:29:53.345590 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:53.345656 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:53.347976 (+ 2320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:53.348829 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:53.348839 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:53.349445 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:53.350102 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:53.350121 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.351465 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.351470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:53.352931 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:53.352945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.353943 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.353953 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:53.354222 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:53.355193 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:53.355230 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:53.358034 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:53.361428 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:53.361437 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:53.361457 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:53.361998 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:53.362449 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:53.362454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:53.362458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:53.362675 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:53.363536 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:53.363543 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:53.364233 (+ 690us) client_negotiation.cc:769] Sending connection context
0807 11:29:53.364713 (+ 480us) client_negotiation.cc:240] Negotiation successful
0807 11:29:53.365355 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":416,"threads_started":1}
I20260807 11:29:53.368654 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:53.343792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:47617 (local address 127.24.103.126:42041)
0807 11:29:53.344382 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:53.344390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:53.345795 (+ 1405us) server_negotiation.cc:407] Connection header received
0807 11:29:53.348302 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:53.348309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:53.348414 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:53.348550 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:53.350324 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.351251 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.353131 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.353647 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.353803 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:53.358341 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:53.358347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:53.358350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:53.358387 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:53.361220 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:53.362156 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:53.362162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:53.362166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:53.362312 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:53.362852 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:53.362861 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:53.362868 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:53.363375 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:53.363560 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:53.368093 (+ 4533us) server_negotiation.cc:299] Negotiation successful
0807 11:29:53.368354 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":290,"threads_started":1}
I20260807 11:29:53.370254 30184 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42041
I20260807 11:29:53.370612 30184 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
I20260807 11:29:53.371486 30184 heartbeater.cc:511] Master 127.24.103.126:42041 requested a full tablet report, sending...
I20260807 11:29:53.373157 29824 ts_manager.cc:194] Registered new tserver with Master: a8d89564cbeb4f98af3a8edf3490b757 (127.24.103.66:46623)
I20260807 11:29:53.374152 29824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:47617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:29:53.415887 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42041
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:29:53.756748 30192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:53.757047 30192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:53.757157 30192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:53.764961 30192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:53.765122 30192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:53.765309 30192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:29:53.777618 30192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:53.779090 30192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:53.780393 30192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:53.793833 30200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:53.793880 30198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:53.793931 30197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:53.796540 30192 server_base.cc:1034] running on GCE node
I20260807 11:29:53.797253 30192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:53.798328 30192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:53.799594 30192 hybrid_clock.cc:648] HybridClock initialized: now 1786102193799555 us; error 75 us; skew 500 ppm
I20260807 11:29:53.799993 30192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:29:53.800166 30039 heartbeater.cc:503] Master 127.24.103.126:42041 was elected leader, sending a full tablet report...
Aug 07 11:29:53 dist-test-slave-wl2h krb5kdc[29772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102193, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:29:53.804081 30192 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:29:53.805437 30192 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:53.805538 30192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:53.805917 30192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:53.809108 30192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca"
format_stamp: "Formatted at 2026-08-07 11:29:53 on dist-test-slave-wl2h"
server_key: "9116dae447dfe488c5322e6e6f91eda9"
server_key_iv: "0c6403059c300316b2e4fbfcd9b8cb42"
server_key_version: "encryptionkey@0"
I20260807 11:29:53.810003 30192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca"
format_stamp: "Formatted at 2026-08-07 11:29:53 on dist-test-slave-wl2h"
server_key: "9116dae447dfe488c5322e6e6f91eda9"
server_key_iv: "0c6403059c300316b2e4fbfcd9b8cb42"
server_key_version: "encryptionkey@0"
I20260807 11:29:53.815821 30192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 11:29:53.819481 30206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:53.821110 30192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:29:53.821307 30192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca"
format_stamp: "Formatted at 2026-08-07 11:29:53 on dist-test-slave-wl2h"
server_key: "9116dae447dfe488c5322e6e6f91eda9"
server_key_iv: "0c6403059c300316b2e4fbfcd9b8cb42"
server_key_version: "encryptionkey@0"
I20260807 11:29:53.821516 30192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:53.841914 30192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:53.847354 30192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:53.847746 30192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:53.849011 30192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:53.850582 30192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:53.850680 30192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:53.850775 30192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:53.850836 30192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:53.889071 30192 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:39575
I20260807 11:29:53.889112 30333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:39575 every 8 connection(s)
I20260807 11:29:53.890082 30192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:29:53.892563 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 30192
I20260807 11:29:53.892798 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:29:53.893373 24989 external_mini_cluster.cc:1428] Setting key bb3cf0ce6df5cea2ef18044445bbc783
Aug 07 11:29:53 dist-test-slave-wl2h krb5kdc[29772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102193, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:29:53.911453 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:53.892978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:39091 (local address 127.24.103.126:42041)
0807 11:29:53.893566 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:53.893572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:53.894693 (+ 1121us) server_negotiation.cc:407] Connection header received
0807 11:29:53.896877 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:53.896881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:53.896991 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:53.897115 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:53.898604 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.899491 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.901091 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.901559 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.901660 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:53.905207 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:53.905211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:53.905215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:53.905250 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:53.908320 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:53.909098 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:53.909105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:53.909108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:53.909233 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:53.909655 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:53.909661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:53.909664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:53.909932 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:53.910096 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:53.910792 (+ 696us) server_negotiation.cc:299] Negotiation successful
0807 11:29:53.911053 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":268,"threads_started":1}
I20260807 11:29:53.912470 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:53.893617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42041 (local address 127.24.103.67:39091)
0807 11:29:53.894535 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:53.894577 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:53.896620 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:53.897335 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:53.897343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:53.897850 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:53.898400 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:53.898416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.899671 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.899675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:53.900949 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:53.900961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.901758 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.901763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:53.901938 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:53.902573 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:29:53.902602 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:53.904992 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:53.908499 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:53.908510 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:53.908528 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:53.908955 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:53.909347 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:29:53.909352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:29:53.909356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:29:53.909540 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:29:53.910077 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:53.910084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:53.910419 (+ 335us) client_negotiation.cc:769] Sending connection context
0807 11:29:53.910700 (+ 281us) client_negotiation.cc:240] Negotiation successful
0807 11:29:53.911284 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":414,"threads_started":1}
I20260807 11:29:53.913931 30335 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42041
I20260807 11:29:53.914306 30335 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:53.915082 30335 heartbeater.cc:511] Master 127.24.103.126:42041 requested a full tablet report, sending...
I20260807 11:29:53.916582 29824 ts_manager.cc:194] Registered new tserver with Master: 12fd0b8b0c5d4228b29a55e65c0d56ca (127.24.103.67:39575)
I20260807 11:29:53.917412 29824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:39091
I20260807 11:29:53.921550 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:29:53.939553 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:53.926441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52224 (local address 127.24.103.126:42041)
0807 11:29:53.926613 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:53.926618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:53.927233 (+ 615us) server_negotiation.cc:407] Connection header received
0807 11:29:53.927369 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:53.927373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:53.927435 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:53.927519 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:53.928971 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.929737 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.931303 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.931704 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.931803 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:53.934123 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:53.934129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:53.934133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:53.934167 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:53.936849 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:53.937515 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:53.937519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:53.937521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:53.937669 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:53.938088 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:53.938093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:53.938095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:53.938449 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:53.938638 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:53.938946 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 11:29:53.939182 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:29:53.943697 29824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:29:53.950539 29824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:29:53.974738 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:53.964859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.24.103.66:46623)
0807 11:29:53.965984 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:53.965992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:53.966015 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:29:53.966094 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:53.966099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:53.966347 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:53.967419 (+ 1072us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:53.967655 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.969192 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.971889 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.973346 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.973504 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:53.973735 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:53.974158 (+ 423us) server_negotiation.cc:299] Negotiation successful
0807 11:29:53.974362 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":855,"threads_started":1}
I20260807 11:29:53.974828 30352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:53.964250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:46623 (local address 127.0.0.1:59306)
0807 11:29:53.965221 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:53.965259 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:53.965442 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:53.966578 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:53.966585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:53.966624 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:53.967230 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:53.967241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.969364 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.969370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:53.971687 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:53.971700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.973590 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.973596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:53.973813 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:53.973875 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 11:29:53.974103 (+ 228us) client_negotiation.cc:240] Negotiation successful
0807 11:29:53.974326 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":97,"thread_start_us":545,"threads_started":1}
I20260807 11:29:53.975205 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:53.965722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37017 (local address 127.0.0.1:37820)
0807 11:29:53.966291 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:53.966309 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:53.966427 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:53.967706 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:53.967711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:53.967744 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:53.968177 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:53.968190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.970105 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.970110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:53.972558 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:53.972573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.974436 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.974442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:53.974620 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:53.974659 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 11:29:53.974735 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 11:29:53.974881 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":806,"threads_started":1}
I20260807 11:29:53.975870 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:53.963796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.24.103.67:39575)
0807 11:29:53.964889 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:53.964898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:53.965846 (+ 948us) server_negotiation.cc:407] Connection header received
0807 11:29:53.965956 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:53.965963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:53.966195 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:53.966392 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:53.967452 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.969192 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.972267 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.973377 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.973549 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:53.973792 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:53.975300 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0807 11:29:53.975490 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":666,"threads_started":1}
I20260807 11:29:53.976269 30350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:53.963674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:39575 (local address 127.0.0.1:36168)
0807 11:29:53.965576 (+ 1902us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:53.965594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:53.965723 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:53.966681 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:53.966686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:53.966733 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:53.967200 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:53.967219 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.969381 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.969385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:53.972038 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:53.972056 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.973919 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.973926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:53.974124 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:53.974159 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:29:53.974262 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 11:29:53.974439 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":77,"spinlock_wait_cycles":3328,"thread_start_us":1506,"threads_started":1}
I20260807 11:29:53.976791 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:53.966295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.24.103.65:37017)
0807 11:29:53.967135 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:53.967141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:53.967163 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:29:53.967271 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:53.967277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:53.967493 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:53.968377 (+ 884us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:53.968632 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.969934 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.972746 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:53.974201 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:53.975918 (+ 1717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:53.976190 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:53.976271 (+ 81us) server_negotiation.cc:299] Negotiation successful
0807 11:29:53.976450 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":483,"threads_started":1}
I20260807 11:29:53.979216 30109 tablet_service.cc:1467] Processing CreateTablet for tablet dd1b217b3ac344b9b638a38080add786 (DEFAULT_TABLE table=test-table [id=ca513ceb8d534e099df67ec8ca99de98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:53.980523 30109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1b217b3ac344b9b638a38080add786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:53.980180 30261 tablet_service.cc:1467] Processing CreateTablet for tablet dd1b217b3ac344b9b638a38080add786 (DEFAULT_TABLE table=test-table [id=ca513ceb8d534e099df67ec8ca99de98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:53.980180 29965 tablet_service.cc:1467] Processing CreateTablet for tablet dd1b217b3ac344b9b638a38080add786 (DEFAULT_TABLE table=test-table [id=ca513ceb8d534e099df67ec8ca99de98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:53.981750 30261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1b217b3ac344b9b638a38080add786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:53.981750 29965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1b217b3ac344b9b638a38080add786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:53.992897 30356 tablet_bootstrap.cc:492] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: Bootstrap starting.
I20260807 11:29:53.996412 30359 tablet_bootstrap.cc:492] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3: Bootstrap starting.
I20260807 11:29:53.996814 30356 tablet_bootstrap.cc:654] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:53.997222 30358 tablet_bootstrap.cc:492] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca: Bootstrap starting.
I20260807 11:29:53.998238 30356 log.cc:824] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:54.000712 30359 tablet_bootstrap.cc:654] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:54.000737 30358 tablet_bootstrap.cc:654] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:54.001685 30359 log.cc:824] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:54.001775 30356 tablet_bootstrap.cc:492] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: No bootstrap required, opened a new log
I20260807 11:29:54.001950 30358 log.cc:824] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:54.002112 30356 ts_tablet_manager.cc:1397] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260807 11:29:54.004830 30359 tablet_bootstrap.cc:492] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3: No bootstrap required, opened a new log
I20260807 11:29:54.004948 30358 tablet_bootstrap.cc:492] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca: No bootstrap required, opened a new log
I20260807 11:29:54.005100 30359 ts_tablet_manager.cc:1397] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260807 11:29:54.005220 30358 ts_tablet_manager.cc:1397] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 11:29:54.012027 30358 raft_consensus.cc:348] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39575 } } peers { permanent_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37017 } } peers { permanent_uuid: "a8d89564cbeb4f98af3a8edf3490b757" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46623 } }
I20260807 11:29:54.012517 30358 raft_consensus.cc:374] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:54.012629 30358 raft_consensus.cc:729] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12fd0b8b0c5d4228b29a55e65c0d56ca, State: Initialized, Role: FOLLOWER
I20260807 11:29:54.012820 30356 raft_consensus.cc:348] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39575 } } peers { permanent_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37017 } } peers { permanent_uuid: "a8d89564cbeb4f98af3a8edf3490b757" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46623 } }
I20260807 11:29:54.013242 30356 raft_consensus.cc:374] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:54.013343 30356 raft_consensus.cc:729] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d89564cbeb4f98af3a8edf3490b757, State: Initialized, Role: FOLLOWER
I20260807 11:29:54.013329 30358 consensus_queue.cc:260] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39575 } } peers { permanent_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37017 } } peers { permanent_uuid: "a8d89564cbeb4f98af3a8edf3490b757" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46623 } }
I20260807 11:29:54.013955 30356 consensus_queue.cc:260] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39575 } } peers { permanent_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37017 } } peers { permanent_uuid: "a8d89564cbeb4f98af3a8edf3490b757" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46623 } }
I20260807 11:29:54.016528 30335 heartbeater.cc:503] Master 127.24.103.126:42041 was elected leader, sending a full tablet report...
I20260807 11:29:54.016389 30359 raft_consensus.cc:348] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39575 } } peers { permanent_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37017 } } peers { permanent_uuid: "a8d89564cbeb4f98af3a8edf3490b757" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46623 } }
I20260807 11:29:54.016922 30359 raft_consensus.cc:374] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:54.017038 30359 raft_consensus.cc:729] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0f7dd1518d418ea84f111fd8b32ae3, State: Initialized, Role: FOLLOWER
I20260807 11:29:54.017289 30358 ts_tablet_manager.cc:1428] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260807 11:29:54.017856 30359 consensus_queue.cc:260] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39575 } } peers { permanent_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37017 } } peers { permanent_uuid: "a8d89564cbeb4f98af3a8edf3490b757" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46623 } }
I20260807 11:29:54.020043 30359 ts_tablet_manager.cc:1428] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260807 11:29:54.021245 30184 heartbeater.cc:503] Master 127.24.103.126:42041 was elected leader, sending a full tablet report...
I20260807 11:29:54.022164 30356 ts_tablet_manager.cc:1428] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260807 11:29:54.030619 30363 raft_consensus.cc:482] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:29:54.030902 30363 raft_consensus.cc:504] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39575 } } peers { permanent_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37017 } } peers { permanent_uuid: "a8d89564cbeb4f98af3a8edf3490b757" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46623 } }
I20260807 11:29:54.032691 30363 leader_election.cc:288] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d0f7dd1518d418ea84f111fd8b32ae3 (127.24.103.65:37017), a8d89564cbeb4f98af3a8edf3490b757 (127.24.103.66:46623)
I20260807 11:29:54.040323 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:54.033014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:35865 (local address 127.24.103.65:37017)
0807 11:29:54.033242 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:54.033248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:54.033533 (+ 285us) server_negotiation.cc:407] Connection header received
0807 11:29:54.033761 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:54.033770 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:54.033865 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:54.033977 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:54.034418 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.035642 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.038222 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.039429 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.039622 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:54.039700 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:54.039961 (+ 261us) server_negotiation.cc:299] Negotiation successful
0807 11:29:54.040094 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:29:54.040467 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:54.033035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37017 (local address 127.24.103.67:35865)
0807 11:29:54.033415 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:54.033437 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:54.033568 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:54.034007 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:54.034014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:54.034050 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:54.034262 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:54.034277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.035843 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.035847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:54.038022 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:54.038050 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.039610 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.039615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:54.039757 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:54.039783 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 11:29:54.039922 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 11:29:54.040077 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":3}
I20260807 11:29:54.040947 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:54.033261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:46623 (local address 127.24.103.67:54327)
0807 11:29:54.034030 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:54.034050 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:54.034236 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:54.034676 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:54.034679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:54.034701 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:29:54.034983 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:54.034991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.036344 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.036352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:54.038362 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:54.038372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.039989 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.039994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:54.040148 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:54.040177 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 11:29:54.040283 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 11:29:54.040401 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":52,"thread_start_us":435,"threads_started":1}
I20260807 11:29:54.041054 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:54.033316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:54327 (local address 127.24.103.66:46623)
0807 11:29:54.033515 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:54.033520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:54.034139 (+ 619us) server_negotiation.cc:407] Connection header received
0807 11:29:54.034423 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:54.034428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:54.034519 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:54.034634 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:29:54.035235 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.036183 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.038512 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.039783 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.039932 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:54.039998 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:54.040318 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 11:29:54.040831 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 11:29:54.041644 29989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1b217b3ac344b9b638a38080add786" candidate_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" is_pre_election: true
I20260807 11:29:54.042001 30131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1b217b3ac344b9b638a38080add786" candidate_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d89564cbeb4f98af3a8edf3490b757" is_pre_election: true
I20260807 11:29:54.042307 29989 raft_consensus.cc:2454] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12fd0b8b0c5d4228b29a55e65c0d56ca in term 0.
I20260807 11:29:54.042476 30131 raft_consensus.cc:2454] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12fd0b8b0c5d4228b29a55e65c0d56ca in term 0.
I20260807 11:29:54.043237 30211 leader_election.cc:302] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12fd0b8b0c5d4228b29a55e65c0d56ca, 3d0f7dd1518d418ea84f111fd8b32ae3, a8d89564cbeb4f98af3a8edf3490b757; no voters:
I20260807 11:29:54.043687 30363 raft_consensus.cc:2781] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:29:54.043855 30363 raft_consensus.cc:482] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:29:54.043978 30363 raft_consensus.cc:3037] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:54.046902 30363 raft_consensus.cc:504] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39575 } } peers { permanent_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37017 } } peers { permanent_uuid: "a8d89564cbeb4f98af3a8edf3490b757" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46623 } }
I20260807 11:29:54.047681 30363 leader_election.cc:288] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [CANDIDATE]: Term 1 election: Requested vote from peers 3d0f7dd1518d418ea84f111fd8b32ae3 (127.24.103.65:37017), a8d89564cbeb4f98af3a8edf3490b757 (127.24.103.66:46623)
I20260807 11:29:54.048117 29989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1b217b3ac344b9b638a38080add786" candidate_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3"
I20260807 11:29:54.048245 30131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1b217b3ac344b9b638a38080add786" candidate_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d89564cbeb4f98af3a8edf3490b757"
I20260807 11:29:54.048350 29989 raft_consensus.cc:3037] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:54.048457 30131 raft_consensus.cc:3037] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:54.051730 29989 raft_consensus.cc:2454] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12fd0b8b0c5d4228b29a55e65c0d56ca in term 1.
I20260807 11:29:54.051730 30131 raft_consensus.cc:2454] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12fd0b8b0c5d4228b29a55e65c0d56ca in term 1.
I20260807 11:29:54.052158 30207 leader_election.cc:302] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12fd0b8b0c5d4228b29a55e65c0d56ca, 3d0f7dd1518d418ea84f111fd8b32ae3; no voters:
I20260807 11:29:54.052486 30363 raft_consensus.cc:2781] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:54.052950 30363 raft_consensus.cc:686] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 1 LEADER]: Becoming Leader. State: Replica: 12fd0b8b0c5d4228b29a55e65c0d56ca, State: Running, Role: LEADER
I20260807 11:29:54.053493 30363 consensus_queue.cc:237] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39575 } } peers { permanent_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37017 } } peers { permanent_uuid: "a8d89564cbeb4f98af3a8edf3490b757" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46623 } }
I20260807 11:29:54.058089 29821 catalog_manager.cc:5495] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 12fd0b8b0c5d4228b29a55e65c0d56ca (127.24.103.67). New cstate: current_term: 1 leader_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12fd0b8b0c5d4228b29a55e65c0d56ca" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8d89564cbeb4f98af3a8edf3490b757" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46623 } health_report { overall_health: UNKNOWN } } }
I20260807 11:29:54.076570 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:54.071220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.24.103.67:39575)
0807 11:29:54.071419 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:54.071423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:54.071587 (+ 164us) server_negotiation.cc:407] Connection header received
0807 11:29:54.071686 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:54.071690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:54.071769 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:54.071901 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:29:54.072283 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.073081 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.074451 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.074858 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.074958 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:54.075511 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:29:54.076050 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:29:54.076192 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:54.076299 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 11:29:54.076368 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 11:29:54.085068 29989 raft_consensus.cc:1261] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3 [term 1 FOLLOWER]: Refusing update from remote peer 12fd0b8b0c5d4228b29a55e65c0d56ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:54.085067 30131 raft_consensus.cc:1261] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757 [term 1 FOLLOWER]: Refusing update from remote peer 12fd0b8b0c5d4228b29a55e65c0d56ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:54.086210 30371 consensus_queue.cc:1035] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d89564cbeb4f98af3a8edf3490b757" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:54.086620 30363 consensus_queue.cc:1035] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0f7dd1518d418ea84f111fd8b32ae3" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 11:29:54.092782 30185 tablet.cc:2369] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:29:54.098408 30375 mvcc.cc:205] Tried to move back new op lower bound from 7315874586961313792 to 7315874586847539200. Current Snapshot: MvccSnapshot[applied={T|T < 7315874586961313792}]
I20260807 11:29:54.100323 30376 mvcc.cc:205] Tried to move back new op lower bound from 7315874586961313792 to 7315874586847539200. Current Snapshot: MvccSnapshot[applied={T|T < 7315874586961313792}]
I20260807 11:29:54.112193 29821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:54.112493 29821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:54.116267 29821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f7cc67b460344b4955569d163794ad0: Sending DeleteTablet for 3 replicas of tablet dd1b217b3ac344b9b638a38080add786
I20260807 11:29:54.117383 30261 tablet_service.cc:1514] Processing DeleteTablet for tablet dd1b217b3ac344b9b638a38080add786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:54 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:36168
I20260807 11:29:54.117568 29965 tablet_service.cc:1514] Processing DeleteTablet for tablet dd1b217b3ac344b9b638a38080add786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:54 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:37820
I20260807 11:29:54.117620 30109 tablet_service.cc:1514] Processing DeleteTablet for tablet dd1b217b3ac344b9b638a38080add786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:54 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:59306
I20260807 11:29:54.119740 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 29896
I20260807 11:29:54.136049 30389 tablet_replica.cc:330] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3: stopping tablet replica
I20260807 11:29:54.136569 30389 raft_consensus.cc:2229] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:54.137095 30389 raft_consensus.cc:2258] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:54.146772 30387 tablet_replica.cc:330] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca: stopping tablet replica
I20260807 11:29:54.149191 30389 ts_tablet_manager.cc:1905] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:54.149902 30387 raft_consensus.cc:2229] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:29:54.152880 30387 raft_consensus.cc:2258] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:54.157770 30389 ts_tablet_manager.cc:1918] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:29:54.158006 30389 log.cc:1196] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/dd1b217b3ac344b9b638a38080add786
I20260807 11:29:54.158296 30387 ts_tablet_manager.cc:1905] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:54.158622 30389 ts_tablet_manager.cc:1939] T dd1b217b3ac344b9b638a38080add786 P 3d0f7dd1518d418ea84f111fd8b32ae3: Deleting consensus metadata
I20260807 11:29:54.159305 30388 tablet_replica.cc:330] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: stopping tablet replica
I20260807 11:29:54.159801 30388 raft_consensus.cc:2229] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:54.160323 30388 raft_consensus.cc:2258] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:54.160764 29803 catalog_manager.cc:4841] TS 3d0f7dd1518d418ea84f111fd8b32ae3 (127.24.103.65:37017): tablet dd1b217b3ac344b9b638a38080add786 (table test-table [id=ca513ceb8d534e099df67ec8ca99de98]) successfully deleted
I20260807 11:29:54.162556 30388 ts_tablet_manager.cc:1905] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:54.167276 30387 ts_tablet_manager.cc:1918] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:29:54.167465 30387 log.cc:1196] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/dd1b217b3ac344b9b638a38080add786
I20260807 11:29:54.168105 30387 ts_tablet_manager.cc:1939] T dd1b217b3ac344b9b638a38080add786 P 12fd0b8b0c5d4228b29a55e65c0d56ca: Deleting consensus metadata
Aug 07 11:29:54 dist-test-slave-wl2h krb5kdc[29772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:29:54.170156 29803 catalog_manager.cc:4841] TS 12fd0b8b0c5d4228b29a55e65c0d56ca (127.24.103.67:39575): tablet dd1b217b3ac344b9b638a38080add786 (table test-table [id=ca513ceb8d534e099df67ec8ca99de98]) successfully deleted
I20260807 11:29:54.174173 30388 ts_tablet_manager.cc:1918] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:29:54.174379 30388 log.cc:1196] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/dd1b217b3ac344b9b638a38080add786
I20260807 11:29:54.174979 30388 ts_tablet_manager.cc:1939] T dd1b217b3ac344b9b638a38080add786 P a8d89564cbeb4f98af3a8edf3490b757: Deleting consensus metadata
I20260807 11:29:54.176388 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:54.121152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.24.103.65:37017)
0807 11:29:54.121381 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:54.121390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:54.142990 (+ 21600us) server_negotiation.cc:407] Connection header received
0807 11:29:54.143642 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:54.143652 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:54.143790 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:54.143883 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:54.147111 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.148636 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.155174 (+ 6538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.156149 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.156295 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:54.169488 (+ 13193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:54.169507 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:54.169526 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:54.169586 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:54.172977 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:54.173750 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:54.173757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:54.173759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:54.173961 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:54.174421 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:54.174433 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:54.174456 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:54.174881 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:54.175267 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:54.175622 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 11:29:54.175869 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:29:54.177132 30032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:54.177124 29805 catalog_manager.cc:4841] TS a8d89564cbeb4f98af3a8edf3490b757 (127.24.103.66:46623): tablet dd1b217b3ac344b9b638a38080add786 (table test-table [id=ca513ceb8d534e099df67ec8ca99de98]) successfully deleted
I20260807 11:29:54.272444 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 29896
I20260807 11:29:54.302271 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 30046
Aug 07 11:29:54 dist-test-slave-wl2h krb5kdc[29772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:29:54.318349 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:54.303210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.24.103.66:46623)
0807 11:29:54.303425 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:54.303431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:54.303451 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:29:54.303602 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:54.303609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:54.303694 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:54.303807 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:54.305606 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.306349 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.307750 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.308140 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.308279 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:54.312569 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:54.312583 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:54.312604 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:54.312642 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:54.315544 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:54.316241 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:54.316248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:54.316251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:54.316379 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:54.316762 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:54.316774 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:54.316776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:54.317128 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:54.317447 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:54.317752 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 11:29:54.317981 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:29:54.319049 30179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:54.409868 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 30046
I20260807 11:29:54.431110 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 30192
Aug 07 11:29:54 dist-test-slave-wl2h krb5kdc[29772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:29:54.452634 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:54.431967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36180 (local address 127.24.103.67:39575)
0807 11:29:54.432193 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:54.432198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:54.432289 (+ 91us) server_negotiation.cc:407] Connection header received
0807 11:29:54.432514 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:54.432519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:54.432603 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:54.432686 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:54.434771 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.435995 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.438090 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:54.438588 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:54.438700 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:54.444691 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:54.444708 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:54.444730 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:29:54.444793 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:54.449432 (+ 4639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:54.450291 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:54.450298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:54.450300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:54.450480 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:29:54.450895 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:29:54.450900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:29:54.450903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:29:54.451352 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:54.451713 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:54.451994 (+ 281us) server_negotiation.cc:299] Negotiation successful
0807 11:29:54.452243 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260807 11:29:54.453333 30329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:54.577930 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 30192
I20260807 11:29:54.602175 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 29787
I20260807 11:29:54.603520 29863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:54.714421 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 29787
2026-08-07T11:29:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4480 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T11:29:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:54Z Disabled control of system clock
I20260807 11:29:54.799860 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:36301
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:36301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 11:29:55.143985 30398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:55.144301 30398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:55.144387 30398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:55.152230 30398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:55.152388 30398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:55.152455 30398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:55.152554 30398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:55.152623 30398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:29:55.162091 30398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:36301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:36301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:55.163619 30398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:55.165481 30398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:55.178748 30406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:55.178912 30404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:55.178719 30403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:55.181030 30398 server_base.cc:1034] running on GCE node
I20260807 11:29:55.181906 30398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:55.183557 30398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:55.184819 30398 hybrid_clock.cc:648] HybridClock initialized: now 1786102195184776 us; error 63 us; skew 500 ppm
I20260807 11:29:55.185243 30398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:29:55.187983 30398 webserver.cc:460] Webserver started at http://127.24.103.126:37347/ using document root <none> and password file <none>
I20260807 11:29:55.188766 30398 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:55.188920 30398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:55.189283 30398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:55.192548 30398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "4f6fcd812e0f48bf8b3c2bfc3bd6e46e"
format_stamp: "Formatted at 2026-08-07 11:29:55 on dist-test-slave-wl2h"
server_key: "1d933a255fd3de593faded4028ab9d7c"
server_key_iv: "12019bdf251d268334c69f9867e0d1e3"
server_key_version: "encryptionkey@0"
I20260807 11:29:55.193403 30398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "4f6fcd812e0f48bf8b3c2bfc3bd6e46e"
format_stamp: "Formatted at 2026-08-07 11:29:55 on dist-test-slave-wl2h"
server_key: "1d933a255fd3de593faded4028ab9d7c"
server_key_iv: "12019bdf251d268334c69f9867e0d1e3"
server_key_version: "encryptionkey@0"
I20260807 11:29:55.198941 30398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 11:29:55.202642 30412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:55.204133 30398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:55.204288 30398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "4f6fcd812e0f48bf8b3c2bfc3bd6e46e"
format_stamp: "Formatted at 2026-08-07 11:29:55 on dist-test-slave-wl2h"
server_key: "1d933a255fd3de593faded4028ab9d7c"
server_key_iv: "12019bdf251d268334c69f9867e0d1e3"
server_key_version: "encryptionkey@0"
I20260807 11:29:55.204473 30398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:55.210451 30398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:55.211336 30398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:55.211603 30398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:55.230650 30398 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:36301
I20260807 11:29:55.230686 30470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:36301 every 8 connection(s)
I20260807 11:29:55.232350 30398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:55.234057 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 30398
I20260807 11:29:55.234254 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:55.234970 24989 external_mini_cluster.cc:1428] Setting key 37b9100f75f9f4731587c76a0281b756
I20260807 11:29:55.236752 30472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:55.244376 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:55.237108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.24.103.126:36301)
0807 11:29:55.238197 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:55.238212 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:55.238270 (+ 58us) server_negotiation.cc:407] Connection header received
0807 11:29:55.240153 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:55.240186 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:55.240772 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:55.241214 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:55.242272 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:55.242280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:55.242301 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:55.242321 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:55.242428 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:55.242451 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:55.242706 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:55.242834 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 11:29:55.243134 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":354,"threads_started":1}
I20260807 11:29:55.250443 30472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e: Bootstrap starting.
I20260807 11:29:55.253767 30472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:55.254732 30472 log.cc:824] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:55.257565 30472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e: No bootstrap required, opened a new log
I20260807 11:29:55.265164 30472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6fcd812e0f48bf8b3c2bfc3bd6e46e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36301 } }
I20260807 11:29:55.265527 30472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:55.265600 30472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6fcd812e0f48bf8b3c2bfc3bd6e46e, State: Initialized, Role: FOLLOWER
I20260807 11:29:55.266095 30472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6fcd812e0f48bf8b3c2bfc3bd6e46e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36301 } }
I20260807 11:29:55.266307 30472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:55.266471 30472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:55.266601 30472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:55.269280 30472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6fcd812e0f48bf8b3c2bfc3bd6e46e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36301 } }
I20260807 11:29:55.269749 30472 leader_election.cc:302] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f6fcd812e0f48bf8b3c2bfc3bd6e46e; no voters:
I20260807 11:29:55.270401 30472 leader_election.cc:288] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:55.270576 30477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:55.271456 30477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [term 1 LEADER]: Becoming Leader. State: Replica: 4f6fcd812e0f48bf8b3c2bfc3bd6e46e, State: Running, Role: LEADER
I20260807 11:29:55.272073 30477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6fcd812e0f48bf8b3c2bfc3bd6e46e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36301 } }
I20260807 11:29:55.272922 30472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:55.275315 30481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f6fcd812e0f48bf8b3c2bfc3bd6e46e. Latest consensus state: current_term: 1 leader_uuid: "4f6fcd812e0f48bf8b3c2bfc3bd6e46e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6fcd812e0f48bf8b3c2bfc3bd6e46e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36301 } } }
I20260807 11:29:55.275368 30480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f6fcd812e0f48bf8b3c2bfc3bd6e46e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6fcd812e0f48bf8b3c2bfc3bd6e46e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 36301 } } }
I20260807 11:29:55.275777 30481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:55.275957 30480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:55.277453 30487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:55.283004 30487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:55.292202 30487 catalog_manager.cc:1347] Generated new cluster ID: ac814cafb29844fa9e27a6cfb11874a6
I20260807 11:29:55.292335 30487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:55.318429 30487 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:55.318970 30487 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:55.329630 30487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e: Generated new TSK 0
I20260807 11:29:55.330785 30487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:29:55.349098 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36301
--builtin_ntp_servers=127.24.103.84:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 11:29:55.696133 30499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:55.696439 30499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:55.696627 30499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:55.704305 30499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:55.704420 30499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:55.704600 30499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:29:55.714749 30499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:55.716310 30499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:55.718367 30499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:55.733338 30505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:55.733647 30507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:55.733356 30504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:55.734273 30499 server_base.cc:1034] running on GCE node
I20260807 11:29:55.734922 30499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:55.736029 30499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:55.737269 30499 hybrid_clock.cc:648] HybridClock initialized: now 1786102195737207 us; error 45 us; skew 500 ppm
I20260807 11:29:55.737591 30499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:29:55.740278 30499 webserver.cc:460] Webserver started at http://127.24.103.65:42209/ using document root <none> and password file <none>
I20260807 11:29:55.740990 30499 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:55.741086 30499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:55.741369 30499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:55.744431 30499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "e21b693be6124bc296a7db03eec0eeb7"
format_stamp: "Formatted at 2026-08-07 11:29:55 on dist-test-slave-wl2h"
server_key: "dc2306b66c115d53d4de910ec8cfac53"
server_key_iv: "502430b89257121d3b8dc5f96655f871"
server_key_version: "encryptionkey@0"
I20260807 11:29:55.745354 30499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "e21b693be6124bc296a7db03eec0eeb7"
format_stamp: "Formatted at 2026-08-07 11:29:55 on dist-test-slave-wl2h"
server_key: "dc2306b66c115d53d4de910ec8cfac53"
server_key_iv: "502430b89257121d3b8dc5f96655f871"
server_key_version: "encryptionkey@0"
I20260807 11:29:55.750869 30499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:29:55.754554 30516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:55.756276 30499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:55.756496 30499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "e21b693be6124bc296a7db03eec0eeb7"
format_stamp: "Formatted at 2026-08-07 11:29:55 on dist-test-slave-wl2h"
server_key: "dc2306b66c115d53d4de910ec8cfac53"
server_key_iv: "502430b89257121d3b8dc5f96655f871"
server_key_version: "encryptionkey@0"
I20260807 11:29:55.756771 30499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:55.763266 30499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:55.764174 30499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:55.764431 30499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:55.765383 30499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:55.766935 30499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:55.767123 30499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:55.767242 30499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:55.767308 30499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:55.805174 30499 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:43605
I20260807 11:29:55.805299 30643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:43605 every 8 connection(s)
I20260807 11:29:55.806543 30499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:29:55.814494 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 30499
I20260807 11:29:55.814666 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:29:55.815400 24989 external_mini_cluster.cc:1428] Setting key f6092c9c463b7779fef4bb24e2e58679
I20260807 11:29:55.816229 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:55.809378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:60863 (local address 127.24.103.126:36301)
0807 11:29:55.809840 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:55.809845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:55.811151 (+ 1306us) server_negotiation.cc:407] Connection header received
0807 11:29:55.812901 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:55.812908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:55.812996 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:55.813121 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:55.815344 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:55.815351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:55.815357 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:55.815384 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:55.815442 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:55.815462 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:55.815645 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:55.815864 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 11:29:55.815931 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":232,"threads_started":1}
I20260807 11:29:55.817395 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:55.810049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36301 (local address 127.24.103.65:60863)
0807 11:29:55.810955 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:55.810990 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:55.812625 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:55.813366 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:55.813375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:55.813466 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:55.814907 (+ 1441us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:29:55.814910 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:55.814951 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:29:55.815156 (+ 205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:55.815646 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:55.815656 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:55.815663 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 11:29:55.815900 (+ 237us) client_negotiation.cc:240] Negotiation successful
0807 11:29:55.816164 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":267,"threads_started":1}
I20260807 11:29:55.818790 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36301
--builtin_ntp_servers=127.24.103.84:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 11:29:55.819103 30644 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36301
I20260807 11:29:55.819465 30644 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:55.820272 30644 heartbeater.cc:511] Master 127.24.103.126:36301 requested a full tablet report, sending...
I20260807 11:29:55.822396 30430 ts_manager.cc:194] Registered new tserver with Master: e21b693be6124bc296a7db03eec0eeb7 (127.24.103.65:43605)
W20260807 11:29:56.156116 30649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:56.156404 30649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:56.156553 30649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:56.163954 30649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:56.164063 30649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:56.164245 30649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:29:56.173995 30649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:56.175513 30649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:56.177353 30649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:56.190697 30655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:56.190784 30654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:56.190761 30657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:56.192051 30649 server_base.cc:1034] running on GCE node
I20260807 11:29:56.192693 30649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:56.193770 30649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:56.195060 30649 hybrid_clock.cc:648] HybridClock initialized: now 1786102196194968 us; error 41 us; skew 500 ppm
I20260807 11:29:56.195427 30649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:29:56.197897 30649 webserver.cc:460] Webserver started at http://127.24.103.66:38993/ using document root <none> and password file <none>
I20260807 11:29:56.198617 30649 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:56.198756 30649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:56.199168 30649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:56.202109 30649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "b71faea174de4a858150cf7dd0f7f5dc"
format_stamp: "Formatted at 2026-08-07 11:29:56 on dist-test-slave-wl2h"
server_key: "ad0316cec74f5feb644d0e2aa7f688a7"
server_key_iv: "1cf51c3c6deb091201cc17dfdf6fad0e"
server_key_version: "encryptionkey@0"
I20260807 11:29:56.203117 30649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "b71faea174de4a858150cf7dd0f7f5dc"
format_stamp: "Formatted at 2026-08-07 11:29:56 on dist-test-slave-wl2h"
server_key: "ad0316cec74f5feb644d0e2aa7f688a7"
server_key_iv: "1cf51c3c6deb091201cc17dfdf6fad0e"
server_key_version: "encryptionkey@0"
I20260807 11:29:56.208448 30649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 11:29:56.212194 30665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:56.213707 30649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:29:56.214004 30649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "b71faea174de4a858150cf7dd0f7f5dc"
format_stamp: "Formatted at 2026-08-07 11:29:56 on dist-test-slave-wl2h"
server_key: "ad0316cec74f5feb644d0e2aa7f688a7"
server_key_iv: "1cf51c3c6deb091201cc17dfdf6fad0e"
server_key_version: "encryptionkey@0"
I20260807 11:29:56.214287 30649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:56.220677 30649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:56.221596 30649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:56.221904 30649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:56.222959 30649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:56.224572 30649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:56.224670 30649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:56.224838 30649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:56.224929 30649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:56.261791 30649 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:40279
I20260807 11:29:56.261816 30791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:40279 every 8 connection(s)
I20260807 11:29:56.263260 30649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:29:56.264214 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 30649
I20260807 11:29:56.264397 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:29:56.264987 24989 external_mini_cluster.cc:1428] Setting key 87293ce4ed6575c14e6724008ddca28d
I20260807 11:29:56.268397 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36301
--builtin_ntp_servers=127.24.103.84:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 11:29:56.273669 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.266307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:60007 (local address 127.24.103.126:36301)
0807 11:29:56.266539 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:56.266546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:56.268368 (+ 1822us) server_negotiation.cc:407] Connection header received
0807 11:29:56.271143 (+ 2775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:56.271147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:56.271216 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:56.271307 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:56.272872 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:56.272877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:56.272880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:56.272898 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:56.272934 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:56.272946 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:56.273336 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:56.273395 (+ 59us) server_negotiation.cc:299] Negotiation successful
0807 11:29:56.273451 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 11:29:56.274914 30794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.266975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36301 (local address 127.24.103.66:60007)
0807 11:29:56.268184 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:56.268233 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:56.270808 (+ 2575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:56.271564 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:56.271577 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:56.271700 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:56.272450 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:29:56.272456 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:56.272514 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:29:56.272647 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:56.273091 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:56.273103 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:56.273113 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 11:29:56.273393 (+ 280us) client_negotiation.cc:240] Negotiation successful
0807 11:29:56.273652 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":355,"threads_started":1}
I20260807 11:29:56.276441 30792 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36301
I20260807 11:29:56.276772 30792 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:56.277428 30792 heartbeater.cc:511] Master 127.24.103.126:36301 requested a full tablet report, sending...
I20260807 11:29:56.279131 30430 ts_manager.cc:194] Registered new tserver with Master: b71faea174de4a858150cf7dd0f7f5dc (127.24.103.66:40279)
W20260807 11:29:56.619647 30795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:56.619956 30795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:56.620119 30795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:56.627674 30795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:56.627799 30795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:56.627981 30795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:29:56.637537 30795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:36301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:56.638958 30795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:56.640851 30795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:56.654764 30801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:56.654819 30802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:56.654737 30804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:56.656263 30795 server_base.cc:1034] running on GCE node
I20260807 11:29:56.656939 30795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:56.658042 30795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:56.659348 30795 hybrid_clock.cc:648] HybridClock initialized: now 1786102196659328 us; error 82 us; skew 500 ppm
I20260807 11:29:56.659706 30795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:29:56.662282 30795 webserver.cc:460] Webserver started at http://127.24.103.67:33127/ using document root <none> and password file <none>
I20260807 11:29:56.663079 30795 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:56.663231 30795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:56.663589 30795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:56.666757 30795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "b056e902db574f1f8303fb4447188e73"
format_stamp: "Formatted at 2026-08-07 11:29:56 on dist-test-slave-wl2h"
server_key: "8d73850b827d20bb9d3e2912aefe490c"
server_key_iv: "cf46f1038bfe61ddfa515070922f2bb5"
server_key_version: "encryptionkey@0"
I20260807 11:29:56.667729 30795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "b056e902db574f1f8303fb4447188e73"
format_stamp: "Formatted at 2026-08-07 11:29:56 on dist-test-slave-wl2h"
server_key: "8d73850b827d20bb9d3e2912aefe490c"
server_key_iv: "cf46f1038bfe61ddfa515070922f2bb5"
server_key_version: "encryptionkey@0"
I20260807 11:29:56.673305 30795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:29:56.677531 30813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:56.679188 30795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:56.679375 30795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "b056e902db574f1f8303fb4447188e73"
format_stamp: "Formatted at 2026-08-07 11:29:56 on dist-test-slave-wl2h"
server_key: "8d73850b827d20bb9d3e2912aefe490c"
server_key_iv: "cf46f1038bfe61ddfa515070922f2bb5"
server_key_version: "encryptionkey@0"
I20260807 11:29:56.679620 30795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:56.685477 30795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:56.686385 30795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:56.686673 30795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:56.687722 30795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:29:56.689247 30795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:29:56.689383 30795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:56.689517 30795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:29:56.689606 30795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:56.726507 30795 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:33505
I20260807 11:29:56.726575 30933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:33505 every 8 connection(s)
I20260807 11:29:56.728024 30795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:29:56.736522 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.730713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:55861 (local address 127.24.103.126:36301)
0807 11:29:56.730913 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:56.730917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:56.732313 (+ 1396us) server_negotiation.cc:407] Connection header received
0807 11:29:56.734030 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:56.734034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:56.734135 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:56.734239 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:56.735768 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:56.735772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:56.735776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:56.735795 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:56.735844 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:56.735857 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:56.735990 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:56.736229 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 11:29:56.736292 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:29:56.737270 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 30795
I20260807 11:29:56.737459 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:29:56.737546 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.731246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:36301 (local address 127.24.103.67:55861)
0807 11:29:56.732154 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:56.732191 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:56.733746 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:56.734465 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:56.734473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:56.734562 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:56.735443 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:29:56.735446 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:56.735484 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:29:56.735573 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:56.735972 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:56.735980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:56.735985 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 11:29:56.736232 (+ 247us) client_negotiation.cc:240] Negotiation successful
0807 11:29:56.736434 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":269,"threads_started":1}
I20260807 11:29:56.738121 24989 external_mini_cluster.cc:1428] Setting key a759af21a8570a91b714033884d46326
I20260807 11:29:56.739441 30934 heartbeater.cc:347] Connected to a master server at 127.24.103.126:36301
I20260807 11:29:56.739774 30934 heartbeater.cc:464] Registering TS with master...
I20260807 11:29:56.740487 30934 heartbeater.cc:511] Master 127.24.103.126:36301 requested a full tablet report, sending...
I20260807 11:29:56.741890 30430 ts_manager.cc:194] Registered new tserver with Master: b056e902db574f1f8303fb4447188e73 (127.24.103.67:33505)
I20260807 11:29:56.753901 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:29:56.761431 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.758710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.24.103.126:36301)
0807 11:29:56.758896 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:56.758901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:56.759407 (+ 506us) server_negotiation.cc:407] Connection header received
0807 11:29:56.759636 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:56.759641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:56.759697 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:56.759809 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:56.760758 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:56.760764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:56.760769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:56.760794 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:56.760827 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:56.760839 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:56.760973 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:56.761060 (+ 87us) server_negotiation.cc:299] Negotiation successful
0807 11:29:56.761170 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:29:56.765795 30430 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:29:56.773239 30430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:29:56.809499 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.787666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:43605 (local address 127.0.0.1:39740)
0807 11:29:56.800528 (+ 12862us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:56.800600 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:56.800813 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:56.805227 (+ 4414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:56.805238 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:56.805337 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:56.807315 (+ 1978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:29:56.807318 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:56.807352 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:29:56.807439 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:56.808692 (+ 1253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:56.808711 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:56.808722 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 11:29:56.808927 (+ 205us) client_negotiation.cc:240] Negotiation successful
0807 11:29:56.809021 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12428,"thread_start_us":15263,"threads_started":1}
I20260807 11:29:56.810374 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.787883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.24.103.65:43605)
0807 11:29:56.802843 (+ 14960us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:56.802855 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:56.802900 (+ 45us) server_negotiation.cc:407] Connection header received
0807 11:29:56.803051 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:56.803065 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:56.803400 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:56.803610 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:56.808218 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:56.808232 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:56.808248 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:56.808275 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:56.808354 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:56.808380 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:56.808620 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:56.809779 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0807 11:29:56.809910 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14739,"thread_start_us":14584,"threads_started":1}
I20260807 11:29:56.810427 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.787911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.24.103.66:40279)
0807 11:29:56.801445 (+ 13534us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:56.801458 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:56.805310 (+ 3852us) server_negotiation.cc:407] Connection header received
0807 11:29:56.806118 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:56.806129 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:56.806384 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:56.806536 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:56.809417 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:56.809425 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:56.809436 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:56.809467 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:56.809524 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:56.809547 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:56.809651 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:56.809953 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 11:29:56.810060 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13355,"thread_start_us":13207,"threads_started":1}
I20260807 11:29:56.810562 30947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.787799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:33505 (local address 127.0.0.1:48794)
0807 11:29:56.803263 (+ 15464us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:56.803288 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:56.803417 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:56.807578 (+ 4161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:56.807585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:56.807616 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:56.808691 (+ 1075us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:29:56.808694 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:56.808714 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:29:56.808909 (+ 195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:56.809941 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:56.809950 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:56.809955 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 11:29:56.810081 (+ 126us) client_negotiation.cc:240] Negotiation successful
0807 11:29:56.810142 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15310,"thread_start_us":13704,"threads_started":1}
I20260807 11:29:56.811723 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.788636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.24.103.67:33505)
0807 11:29:56.806134 (+ 17498us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:56.806143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:56.806193 (+ 50us) server_negotiation.cc:407] Connection header received
0807 11:29:56.806264 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:56.806271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:56.806453 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:56.806596 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:56.809286 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:56.809294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:56.809377 (+ 83us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:56.809400 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:56.809456 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:56.809480 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:56.811150 (+ 1670us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:56.811230 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 11:29:56.811326 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16974,"thread_start_us":16783,"threads_started":1}
I20260807 11:29:56.812387 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.788085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:40279 (local address 127.0.0.1:58930)
0807 11:29:56.805166 (+ 17081us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:56.805195 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:56.805893 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:56.807657 (+ 1764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:56.807663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:56.807697 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:56.809144 (+ 1447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:29:56.809148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:56.809163 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:29:56.809220 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:56.809746 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:56.809750 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:56.809754 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 11:29:56.810696 (+ 942us) client_negotiation.cc:240] Negotiation successful
0807 11:29:56.810748 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16932,"thread_start_us":16791,"threads_started":1}
I20260807 11:29:56.814698 30569 tablet_service.cc:1467] Processing CreateTablet for tablet a4b31c81dbf34a309f00bbf8e08f9866 (DEFAULT_TABLE table=test-table [id=804b716875ea4e79870f6a3bb7536ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:56.816375 30569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b31c81dbf34a309f00bbf8e08f9866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:56.815999 30868 tablet_service.cc:1467] Processing CreateTablet for tablet a4b31c81dbf34a309f00bbf8e08f9866 (DEFAULT_TABLE table=test-table [id=804b716875ea4e79870f6a3bb7536ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:56.816004 30721 tablet_service.cc:1467] Processing CreateTablet for tablet a4b31c81dbf34a309f00bbf8e08f9866 (DEFAULT_TABLE table=test-table [id=804b716875ea4e79870f6a3bb7536ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:29:56.817404 30868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b31c81dbf34a309f00bbf8e08f9866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:56.817404 30721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b31c81dbf34a309f00bbf8e08f9866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:56.825436 30644 heartbeater.cc:503] Master 127.24.103.126:36301 was elected leader, sending a full tablet report...
I20260807 11:29:56.830245 30954 tablet_bootstrap.cc:492] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7: Bootstrap starting.
I20260807 11:29:56.832382 30956 tablet_bootstrap.cc:492] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73: Bootstrap starting.
I20260807 11:29:56.834332 30954 tablet_bootstrap.cc:654] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:56.836043 30954 log.cc:824] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:56.836302 30956 tablet_bootstrap.cc:654] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:56.837656 30956 log.cc:824] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:56.837742 30955 tablet_bootstrap.cc:492] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc: Bootstrap starting.
I20260807 11:29:56.839923 30954 tablet_bootstrap.cc:492] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7: No bootstrap required, opened a new log
I20260807 11:29:56.840260 30954 ts_tablet_manager.cc:1397] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 11:29:56.842232 30956 tablet_bootstrap.cc:492] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73: No bootstrap required, opened a new log
I20260807 11:29:56.842254 30955 tablet_bootstrap.cc:654] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:56.842519 30956 ts_tablet_manager.cc:1397] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 11:29:56.843463 30955 log.cc:824] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:56.849414 30955 tablet_bootstrap.cc:492] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc: No bootstrap required, opened a new log
I20260807 11:29:56.849720 30955 ts_tablet_manager.cc:1397] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260807 11:29:56.851462 30954 raft_consensus.cc:348] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21b693be6124bc296a7db03eec0eeb7" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43605 } } peers { permanent_uuid: "b71faea174de4a858150cf7dd0f7f5dc" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40279 } } peers { permanent_uuid: "b056e902db574f1f8303fb4447188e73" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 33505 } }
I20260807 11:29:56.851935 30954 raft_consensus.cc:374] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:56.852125 30954 raft_consensus.cc:729] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e21b693be6124bc296a7db03eec0eeb7, State: Initialized, Role: FOLLOWER
I20260807 11:29:56.852813 30954 consensus_queue.cc:260] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21b693be6124bc296a7db03eec0eeb7" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43605 } } peers { permanent_uuid: "b71faea174de4a858150cf7dd0f7f5dc" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40279 } } peers { permanent_uuid: "b056e902db574f1f8303fb4447188e73" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 33505 } }
I20260807 11:29:56.854463 30956 raft_consensus.cc:348] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21b693be6124bc296a7db03eec0eeb7" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43605 } } peers { permanent_uuid: "b71faea174de4a858150cf7dd0f7f5dc" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40279 } } peers { permanent_uuid: "b056e902db574f1f8303fb4447188e73" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 33505 } }
I20260807 11:29:56.855223 30956 raft_consensus.cc:374] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:56.855352 30956 raft_consensus.cc:729] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b056e902db574f1f8303fb4447188e73, State: Initialized, Role: FOLLOWER
I20260807 11:29:56.856352 30956 consensus_queue.cc:260] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21b693be6124bc296a7db03eec0eeb7" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43605 } } peers { permanent_uuid: "b71faea174de4a858150cf7dd0f7f5dc" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40279 } } peers { permanent_uuid: "b056e902db574f1f8303fb4447188e73" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 33505 } }
I20260807 11:29:56.857750 30934 heartbeater.cc:503] Master 127.24.103.126:36301 was elected leader, sending a full tablet report...
I20260807 11:29:56.858733 30956 ts_tablet_manager.cc:1428] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260807 11:29:56.859822 30955 raft_consensus.cc:348] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21b693be6124bc296a7db03eec0eeb7" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43605 } } peers { permanent_uuid: "b71faea174de4a858150cf7dd0f7f5dc" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40279 } } peers { permanent_uuid: "b056e902db574f1f8303fb4447188e73" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 33505 } }
I20260807 11:29:56.860271 30955 raft_consensus.cc:374] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:56.860433 30955 raft_consensus.cc:729] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71faea174de4a858150cf7dd0f7f5dc, State: Initialized, Role: FOLLOWER
I20260807 11:29:56.861181 30955 consensus_queue.cc:260] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21b693be6124bc296a7db03eec0eeb7" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43605 } } peers { permanent_uuid: "b71faea174de4a858150cf7dd0f7f5dc" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40279 } } peers { permanent_uuid: "b056e902db574f1f8303fb4447188e73" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 33505 } }
I20260807 11:29:56.861450 30954 ts_tablet_manager.cc:1428] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7: Time spent starting tablet: real 0.021s user 0.010s sys 0.011s
I20260807 11:29:56.862982 30792 heartbeater.cc:503] Master 127.24.103.126:36301 was elected leader, sending a full tablet report...
I20260807 11:29:56.863883 30955 ts_tablet_manager.cc:1428] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260807 11:29:56.864259 30961 raft_consensus.cc:482] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:29:56.864579 30961 raft_consensus.cc:504] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21b693be6124bc296a7db03eec0eeb7" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43605 } } peers { permanent_uuid: "b71faea174de4a858150cf7dd0f7f5dc" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40279 } } peers { permanent_uuid: "b056e902db574f1f8303fb4447188e73" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 33505 } }
I20260807 11:29:56.866463 30961 leader_election.cc:288] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b71faea174de4a858150cf7dd0f7f5dc (127.24.103.66:40279), b056e902db574f1f8303fb4447188e73 (127.24.103.67:33505)
I20260807 11:29:56.870432 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.867426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:53665 (local address 127.24.103.66:40279)
0807 11:29:56.867602 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:56.867610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:56.867724 (+ 114us) server_negotiation.cc:407] Connection header received
0807 11:29:56.867968 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:56.867973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:56.868034 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:56.868105 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:56.869321 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:56.869330 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:56.869337 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:56.869363 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:56.869406 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:56.869426 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:56.869525 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:56.869797 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 11:29:56.869863 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 11:29:56.871217 30964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.866833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:40279 (local address 127.24.103.65:53665)
0807 11:29:56.867631 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:56.867651 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:56.867815 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:56.868279 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:56.868282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:56.868302 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:56.869074 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:29:56.869077 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:56.869096 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:29:56.869147 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:56.869641 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:56.869646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:56.869651 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 11:29:56.870717 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0807 11:29:56.870768 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":1029,"threads_started":1}
I20260807 11:29:56.872310 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.867872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:37661 (local address 127.24.103.67:33505)
0807 11:29:56.868085 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:56.868094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:56.869844 (+ 1750us) server_negotiation.cc:407] Connection header received
0807 11:29:56.869972 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:56.869979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:56.870044 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:56.870176 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:56.871599 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:56.871606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:56.871611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:56.871632 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:56.871674 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:56.871688 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:56.871836 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:56.871922 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 11:29:56.871996 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:29:56.872187 30743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b31c81dbf34a309f00bbf8e08f9866" candidate_uuid: "e21b693be6124bc296a7db03eec0eeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b71faea174de4a858150cf7dd0f7f5dc" is_pre_election: true
I20260807 11:29:56.872505 30965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.867288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:33505 (local address 127.24.103.65:37661)
0807 11:29:56.869734 (+ 2446us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:56.869753 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:56.869862 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:56.870478 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:56.870482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:56.870503 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:29:56.871360 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:29:56.871363 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:29:56.871383 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:29:56.871433 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:29:56.871796 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:29:56.871801 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:29:56.871808 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 11:29:56.871967 (+ 159us) client_negotiation.cc:240] Negotiation successful
0807 11:29:56.872039 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2311,"thread_start_us":2111,"threads_started":1}
I20260807 11:29:56.872642 30743 raft_consensus.cc:2454] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e21b693be6124bc296a7db03eec0eeb7 in term 0.
I20260807 11:29:56.873344 30517 leader_election.cc:302] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b71faea174de4a858150cf7dd0f7f5dc, e21b693be6124bc296a7db03eec0eeb7; no voters:
I20260807 11:29:56.873546 30889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b31c81dbf34a309f00bbf8e08f9866" candidate_uuid: "e21b693be6124bc296a7db03eec0eeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b056e902db574f1f8303fb4447188e73" is_pre_election: true
I20260807 11:29:56.873812 30961 raft_consensus.cc:2781] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:29:56.873970 30961 raft_consensus.cc:482] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:29:56.874084 30961 raft_consensus.cc:3037] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:56.874182 30889 raft_consensus.cc:2454] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e21b693be6124bc296a7db03eec0eeb7 in term 0.
I20260807 11:29:56.877099 30961 raft_consensus.cc:504] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21b693be6124bc296a7db03eec0eeb7" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43605 } } peers { permanent_uuid: "b71faea174de4a858150cf7dd0f7f5dc" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40279 } } peers { permanent_uuid: "b056e902db574f1f8303fb4447188e73" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 33505 } }
I20260807 11:29:56.877830 30961 leader_election.cc:288] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [CANDIDATE]: Term 1 election: Requested vote from peers b71faea174de4a858150cf7dd0f7f5dc (127.24.103.66:40279), b056e902db574f1f8303fb4447188e73 (127.24.103.67:33505)
I20260807 11:29:56.878420 30743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b31c81dbf34a309f00bbf8e08f9866" candidate_uuid: "e21b693be6124bc296a7db03eec0eeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b71faea174de4a858150cf7dd0f7f5dc"
I20260807 11:29:56.878463 30889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b31c81dbf34a309f00bbf8e08f9866" candidate_uuid: "e21b693be6124bc296a7db03eec0eeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b056e902db574f1f8303fb4447188e73"
I20260807 11:29:56.878679 30743 raft_consensus.cc:3037] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:56.878716 30889 raft_consensus.cc:3037] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:56.881152 30743 raft_consensus.cc:2454] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e21b693be6124bc296a7db03eec0eeb7 in term 1.
I20260807 11:29:56.881342 30889 raft_consensus.cc:2454] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e21b693be6124bc296a7db03eec0eeb7 in term 1.
I20260807 11:29:56.881654 30517 leader_election.cc:302] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b71faea174de4a858150cf7dd0f7f5dc, e21b693be6124bc296a7db03eec0eeb7; no voters:
I20260807 11:29:56.881990 30961 raft_consensus.cc:2781] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:56.882489 30961 raft_consensus.cc:686] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [term 1 LEADER]: Becoming Leader. State: Replica: e21b693be6124bc296a7db03eec0eeb7, State: Running, Role: LEADER
I20260807 11:29:56.883118 30961 consensus_queue.cc:237] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21b693be6124bc296a7db03eec0eeb7" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43605 } } peers { permanent_uuid: "b71faea174de4a858150cf7dd0f7f5dc" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40279 } } peers { permanent_uuid: "b056e902db574f1f8303fb4447188e73" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 33505 } }
I20260807 11:29:56.888341 30430 catalog_manager.cc:5495] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e21b693be6124bc296a7db03eec0eeb7 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "e21b693be6124bc296a7db03eec0eeb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21b693be6124bc296a7db03eec0eeb7" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 43605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b71faea174de4a858150cf7dd0f7f5dc" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b056e902db574f1f8303fb4447188e73" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 33505 } health_report { overall_health: UNKNOWN } } }
I20260807 11:29:56.916432 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.913765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.24.103.65:43605)
0807 11:29:56.913988 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:56.913995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:56.914179 (+ 184us) server_negotiation.cc:407] Connection header received
0807 11:29:56.914366 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:56.914370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:56.914471 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:56.914575 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:56.915705 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:56.915710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:56.915713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:56.915732 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:56.915768 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:56.915780 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:56.915968 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:56.916032 (+ 64us) server_negotiation.cc:299] Negotiation successful
0807 11:29:56.916129 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:29:56.925698 30743 raft_consensus.cc:1261] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc [term 1 FOLLOWER]: Refusing update from remote peer e21b693be6124bc296a7db03eec0eeb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:56.925693 30889 raft_consensus.cc:1261] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73 [term 1 FOLLOWER]: Refusing update from remote peer e21b693be6124bc296a7db03eec0eeb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:29:56.926717 30966 consensus_queue.cc:1035] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b71faea174de4a858150cf7dd0f7f5dc" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:56.927143 30961 consensus_queue.cc:1035] T a4b31c81dbf34a309f00bbf8e08f9866 P e21b693be6124bc296a7db03eec0eeb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b056e902db574f1f8303fb4447188e73" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:29:56.940227 30971 mvcc.cc:205] Tried to move back new op lower bound from 7315874598597095424 to 7315874598437699584. Current Snapshot: MvccSnapshot[applied={T|T < 7315874598597095424}]
I20260807 11:29:56.941860 30973 mvcc.cc:205] Tried to move back new op lower bound from 7315874598597095424 to 7315874598437699584. Current Snapshot: MvccSnapshot[applied={T|T < 7315874598597095424}]
I20260807 11:29:56.956542 30430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:56.956883 30430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:29:56.961370 30430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f6fcd812e0f48bf8b3c2bfc3bd6e46e: Sending DeleteTablet for 3 replicas of tablet a4b31c81dbf34a309f00bbf8e08f9866
I20260807 11:29:56.962809 30868 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b31c81dbf34a309f00bbf8e08f9866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:56 UTC) from {username='slave'} at 127.0.0.1:48794
I20260807 11:29:56.962833 30569 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b31c81dbf34a309f00bbf8e08f9866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:56 UTC) from {username='slave'} at 127.0.0.1:39740
I20260807 11:29:56.962833 30721 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b31c81dbf34a309f00bbf8e08f9866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:29:56 UTC) from {username='slave'} at 127.0.0.1:58930
I20260807 11:29:56.968230 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 30499
I20260807 11:29:56.978117 30984 tablet_replica.cc:330] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73: stopping tablet replica
I20260807 11:29:56.978777 30984 raft_consensus.cc:2229] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:56.979472 30984 raft_consensus.cc:2258] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:56.980736 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:56.969805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.24.103.65:43605)
0807 11:29:56.970028 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:56.970035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:56.971987 (+ 1952us) server_negotiation.cc:407] Connection header received
0807 11:29:56.973383 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:56.973394 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:56.973497 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:56.973602 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:56.976721 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:56.976742 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:56.976750 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:56.976785 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:56.976844 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:56.976867 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:56.977034 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:56.980170 (+ 3136us) server_negotiation.cc:299] Negotiation successful
0807 11:29:56.980275 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:29:56.982385 30637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:56.982450 30984 ts_tablet_manager.cc:1905] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:56.984156 30985 tablet_replica.cc:330] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc: stopping tablet replica
I20260807 11:29:56.984664 30985 raft_consensus.cc:2229] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:29:56.985227 30985 raft_consensus.cc:2258] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:29:56.987700 30985 ts_tablet_manager.cc:1905] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:29:56.990772 30984 ts_tablet_manager.cc:1918] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:29:56.991084 30984 log.cc:1196] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/a4b31c81dbf34a309f00bbf8e08f9866
I20260807 11:29:56.991772 30984 ts_tablet_manager.cc:1939] T a4b31c81dbf34a309f00bbf8e08f9866 P b056e902db574f1f8303fb4447188e73: Deleting consensus metadata
I20260807 11:29:56.993891 30416 catalog_manager.cc:4841] TS b056e902db574f1f8303fb4447188e73 (127.24.103.67:33505): tablet a4b31c81dbf34a309f00bbf8e08f9866 (table test-table [id=804b716875ea4e79870f6a3bb7536ce3]) successfully deleted
I20260807 11:29:56.994699 30985 ts_tablet_manager.cc:1918] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:29:56.994872 30985 log.cc:1196] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/a4b31c81dbf34a309f00bbf8e08f9866
I20260807 11:29:56.995358 30985 ts_tablet_manager.cc:1939] T a4b31c81dbf34a309f00bbf8e08f9866 P b71faea174de4a858150cf7dd0f7f5dc: Deleting consensus metadata
I20260807 11:29:56.996842 30413 catalog_manager.cc:4841] TS b71faea174de4a858150cf7dd0f7f5dc (127.24.103.66:40279): tablet a4b31c81dbf34a309f00bbf8e08f9866 (table test-table [id=804b716875ea4e79870f6a3bb7536ce3]) successfully deleted
I20260807 11:29:57.086480 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 30499
I20260807 11:29:57.108186 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 30649
W20260807 11:29:57.108304 30415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:43605 (error 108)
I20260807 11:29:57.110546 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:57.109937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:43605 (local address 127.0.0.1:39762)
0807 11:29:57.110111 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:57.110341 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260807 11:29:57.110867 30415 catalog_manager.cc:4584] TS e21b693be6124bc296a7db03eec0eeb7 (127.24.103.65:43605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4b31c81dbf34a309f00bbf8e08f9866: Network error: Client connection negotiation failed: client connection to 127.24.103.65:43605: connect: Connection refused (error 111)
I20260807 11:29:57.112058 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:57.109483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.24.103.66:40279)
0807 11:29:57.109794 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:57.109804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:57.109835 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:29:57.109918 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:57.109924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:57.110044 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:57.110172 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:57.111353 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:57.111361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:57.111367 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:57.111387 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:57.111430 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:57.111444 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:57.111551 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:57.111672 (+ 121us) server_negotiation.cc:299] Negotiation successful
0807 11:29:57.111727 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 11:29:57.112759 30787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:57.165082 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:57.164346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:43605 (local address 127.0.0.1:39768)
0807 11:29:57.164761 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:57.164885 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260807 11:29:57.166607 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:57.166123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:43605 (local address 127.0.0.1:39770)
0807 11:29:57.166370 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:57.166476 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260807 11:29:57.206303 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 30649
I20260807 11:29:57.226655 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 30795
I20260807 11:29:57.227587 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:57.226782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:43605 (local address 127.0.0.1:39786)
0807 11:29:57.227226 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:57.227371 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260807 11:29:57.228863 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:57.228378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:43605 (local address 127.0.0.1:39788)
0807 11:29:57.228597 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:57.228693 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 11:29:57.230156 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:57.227801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.24.103.67:33505)
0807 11:29:57.228001 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:57.228005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:57.228023 (+ 18us) server_negotiation.cc:407] Connection header received
0807 11:29:57.228078 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:57.228082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:57.228218 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:57.228345 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:57.229447 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:57.229453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:57.229456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:57.229475 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:57.229509 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:57.229521 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:57.229649 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:57.229782 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 11:29:57.229867 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 11:29:57.230926 30929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:57.309705 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:57.309150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:43605 (local address 127.0.0.1:39804)
0807 11:29:57.309392 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:57.309517 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 11:29:57.311231 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:57.310610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:43605 (local address 127.0.0.1:39820)
0807 11:29:57.310907 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:57.311002 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260807 11:29:57.322582 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 30795
I20260807 11:29:57.346709 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 30398
I20260807 11:29:57.348361 30464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:57.465143 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 30398
2026-08-07T11:29:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2746 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 11:29:57.511475 24989 mini_oidc.cc:152] Starting JWKS server
I20260807 11:29:57.514420 24989 webserver.cc:460] Webserver started at https://127.0.0.1:35391/ using document root <none> and password file <none>
I20260807 11:29:57.514696 24989 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 11:29:57.515393 24989 webserver.cc:460] Webserver started at http://127.0.0.1:32921/ using document root <none> and password file <none>
2026-08-07T11:29:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:57Z Disabled control of system clock
I20260807 11:29:57.551493 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41317
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:41317
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35391/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/testchainca.cert with env {}
W20260807 11:29:57.892318 31002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:57.892678 31002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:57.892839 31002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:57.900238 31002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:57.900367 31002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:57.900406 31002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:57.900456 31002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:57.900503 31002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 11:29:57.900568 31002 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 11:29:57.900607 31002 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35391/jwks.json
I20260807 11:29:57.910779 31002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:41317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35391/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:57.912222 31002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:57.913583 31002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:57.925534 31009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:57.925617 31008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:57.925647 31013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:57.927378 31002 server_base.cc:1034] running on GCE node
I20260807 11:29:57.928211 31002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:57.929749 31002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:57.930995 31002 hybrid_clock.cc:648] HybridClock initialized: now 1786102197931004 us; error 101 us; skew 500 ppm
I20260807 11:29:57.931438 31002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:29:57.933979 31002 webserver.cc:460] Webserver started at http://127.24.103.126:44763/ using document root <none> and password file <none>
I20260807 11:29:57.934722 31002 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:57.934868 31002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:57.935243 31002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:57.938297 31002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "684f3225b0814c59aedf7e972c7ffe58"
format_stamp: "Formatted at 2026-08-07 11:29:57 on dist-test-slave-wl2h"
server_key: "b415d15b0e45ad3ba05bf60a74c71b06"
server_key_iv: "f43a771ec8666867453bb361324dcb5b"
server_key_version: "encryptionkey@0"
I20260807 11:29:57.939254 31002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "684f3225b0814c59aedf7e972c7ffe58"
format_stamp: "Formatted at 2026-08-07 11:29:57 on dist-test-slave-wl2h"
server_key: "b415d15b0e45ad3ba05bf60a74c71b06"
server_key_iv: "f43a771ec8666867453bb361324dcb5b"
server_key_version: "encryptionkey@0"
I20260807 11:29:57.944630 31002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:29:57.948266 31020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:57.949661 31002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:29:57.949882 31002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "684f3225b0814c59aedf7e972c7ffe58"
format_stamp: "Formatted at 2026-08-07 11:29:57 on dist-test-slave-wl2h"
server_key: "b415d15b0e45ad3ba05bf60a74c71b06"
server_key_iv: "f43a771ec8666867453bb361324dcb5b"
server_key_version: "encryptionkey@0"
I20260807 11:29:57.950120 31002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:57.966830 31002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:57.967741 31002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:57.968067 31002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:57.987732 31002 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:41317
I20260807 11:29:57.987757 31078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:41317 every 8 connection(s)
I20260807 11:29:57.989141 31002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:57.992865 31079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:57.995276 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 31002
I20260807 11:29:57.995500 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:57.996316 24989 external_mini_cluster.cc:1428] Setting key 9e3ffb71246f87118a71dc205eed312c
I20260807 11:29:58.005579 31079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58: Bootstrap starting.
I20260807 11:29:58.009423 31079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:58.010618 31079 log.cc:824] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:58.011327 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:57.998401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.24.103.126:41317)
0807 11:29:57.999471 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:57.999482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:57.999533 (+ 51us) server_negotiation.cc:407] Connection header received
0807 11:29:58.001232 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:58.001255 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:58.001730 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:58.002068 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:58.002769 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.004186 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.006119 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.006825 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.006970 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:58.008491 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:58.008499 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:58.008513 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:58.009115 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:58.009233 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:58.009262 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:58.009364 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:58.009718 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 11:29:58.010114 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":418,"threads_started":1}
I20260807 11:29:58.014410 31079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58: No bootstrap required, opened a new log
I20260807 11:29:58.016448 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:29:58.023990 31079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684f3225b0814c59aedf7e972c7ffe58" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41317 } }
I20260807 11:29:58.024363 31079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:58.024513 31079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684f3225b0814c59aedf7e972c7ffe58, State: Initialized, Role: FOLLOWER
I20260807 11:29:58.025084 31079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684f3225b0814c59aedf7e972c7ffe58" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41317 } }
I20260807 11:29:58.025342 31079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:58.025491 31079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:58.025668 31079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:58.028227 31079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684f3225b0814c59aedf7e972c7ffe58" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41317 } }
I20260807 11:29:58.028718 31079 leader_election.cc:302] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 684f3225b0814c59aedf7e972c7ffe58; no voters:
I20260807 11:29:58.029349 31079 leader_election.cc:288] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:58.029534 31084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:58.030627 31084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [term 1 LEADER]: Becoming Leader. State: Replica: 684f3225b0814c59aedf7e972c7ffe58, State: Running, Role: LEADER
I20260807 11:29:58.031234 31084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684f3225b0814c59aedf7e972c7ffe58" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41317 } }
I20260807 11:29:58.031690 31079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:58.034016 31086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 684f3225b0814c59aedf7e972c7ffe58. Latest consensus state: current_term: 1 leader_uuid: "684f3225b0814c59aedf7e972c7ffe58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684f3225b0814c59aedf7e972c7ffe58" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41317 } } }
I20260807 11:29:58.034413 31086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:58.034940 31085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "684f3225b0814c59aedf7e972c7ffe58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684f3225b0814c59aedf7e972c7ffe58" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41317 } } }
I20260807 11:29:58.035310 31085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58 [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:58.035638 31094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:58.047609 31094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:58.057214 31094 catalog_manager.cc:1347] Generated new cluster ID: 75396262003b41ac8213f74514a7b10f
I20260807 11:29:58.057345 31094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:58.097313 31094 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:58.098645 31094 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:58.119364 31094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 684f3225b0814c59aedf7e972c7ffe58: Generated new TSK 0
I20260807 11:29:58.120250 31094 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 11:29:58.171814 31115 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 11:29:58.172075 31115 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 11:29:58.173025 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:58.142473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48782 (local address 127.24.103.126:41317)
0807 11:29:58.142747 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:58.142756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:58.143276 (+ 520us) server_negotiation.cc:407] Connection header received
0807 11:29:58.143762 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:58.143767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:58.143843 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:58.143959 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0807 11:29:58.144695 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.145490 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.147732 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.148286 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.148424 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:58.149124 (+ 700us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 11:29:58.172327 (+ 23203us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 11:29:58.172664 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48782: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":79}
W20260807 11:29:58.173221 31082 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48782: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 11:29:58.175344 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:29:58.142376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41317 (local address 127.0.0.1:48782)
0807 11:29:58.143153 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:58.143176 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:58.143388 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:58.144028 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:58.144034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:58.144047 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0807 11:29:58.144492 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:58.144506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.145664 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.145671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:58.147548 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:58.147574 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.148505 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.148512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:58.148718 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:58.148819 (+ 101us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 11:29:58.172692 (+ 23873us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 11:29:58.173238 (+ 546us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.103.126:41317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":356,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.103.126:41317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 11:29:58.178112 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 31002
I20260807 11:29:58.179582 31073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:58.282145 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 31002
2026-08-07T11:29:58Z chronyd exiting
I20260807 11:29:58.338830 24989 test_util.cc:175] -----------------------------------------------
I20260807 11:29:58.339058 24989 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786102153172159-24989-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (832 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 11:29:58.343232 24989 mini_oidc.cc:152] Starting JWKS server
I20260807 11:29:58.345289 24989 webserver.cc:460] Webserver started at https://127.0.0.1:39253/ using document root <none> and password file <none>
I20260807 11:29:58.345567 24989 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 11:29:58.346294 24989 webserver.cc:460] Webserver started at http://127.0.0.1:37037/ using document root <none> and password file <none>
2026-08-07T11:29:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:58Z Disabled control of system clock
I20260807 11:29:58.383373 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41639
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:41639
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39253/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/testchainca.cert with env {}
W20260807 11:29:58.723361 31124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:58.723658 31124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:58.723843 31124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:58.731305 31124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:58.731410 31124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:58.731503 31124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:58.731618 31124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:58.731706 31124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 11:29:58.731798 31124 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 11:29:58.731858 31124 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39253/jwks.json
I20260807 11:29:58.742175 31124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:41639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39253/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:58.743609 31124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:58.744987 31124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:58.757664 31132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:58.757634 31130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:58.757611 31129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:58.758802 31124 server_base.cc:1034] running on GCE node
I20260807 11:29:58.759649 31124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:58.761080 31124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:58.762353 31124 hybrid_clock.cc:648] HybridClock initialized: now 1786102198762293 us; error 54 us; skew 500 ppm
I20260807 11:29:58.762722 31124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:29:58.765107 31124 webserver.cc:460] Webserver started at http://127.24.103.126:34303/ using document root <none> and password file <none>
I20260807 11:29:58.765837 31124 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:58.765933 31124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:58.766201 31124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:58.769116 31124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "13745572500c4cabaa3e2b02136d36fa"
format_stamp: "Formatted at 2026-08-07 11:29:58 on dist-test-slave-wl2h"
server_key: "15d09266ca1ddae130b925c8358b39be"
server_key_iv: "d148c7ea0921ee35c806eafda10c52ca"
server_key_version: "encryptionkey@0"
I20260807 11:29:58.769876 31124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "13745572500c4cabaa3e2b02136d36fa"
format_stamp: "Formatted at 2026-08-07 11:29:58 on dist-test-slave-wl2h"
server_key: "15d09266ca1ddae130b925c8358b39be"
server_key_iv: "d148c7ea0921ee35c806eafda10c52ca"
server_key_version: "encryptionkey@0"
I20260807 11:29:58.775409 31124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 11:29:58.779287 31139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:58.780745 31124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:29:58.780892 31124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "13745572500c4cabaa3e2b02136d36fa"
format_stamp: "Formatted at 2026-08-07 11:29:58 on dist-test-slave-wl2h"
server_key: "15d09266ca1ddae130b925c8358b39be"
server_key_iv: "d148c7ea0921ee35c806eafda10c52ca"
server_key_version: "encryptionkey@0"
I20260807 11:29:58.781080 31124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:58.824239 31124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:58.825081 31124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:58.825373 31124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:58.845690 31124 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:41639
I20260807 11:29:58.845739 31192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:41639 every 8 connection(s)
I20260807 11:29:58.847174 31124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:58.851540 31193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:58.857429 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 31124
I20260807 11:29:58.857663 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:58.858502 24989 external_mini_cluster.cc:1428] Setting key 3ffab84ce037f0cb1a930fe21fa11394
I20260807 11:29:58.864614 31193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa: Bootstrap starting.
I20260807 11:29:58.868386 31193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:58.869663 31193 log.cc:824] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:58.873287 31196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:58.860687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.24.103.126:41639)
0807 11:29:58.861756 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:58.861768 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:58.861817 (+ 49us) server_negotiation.cc:407] Connection header received
0807 11:29:58.863658 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:58.863682 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:58.864156 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:58.864474 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:58.865135 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.866486 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.868391 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.869131 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.869272 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:58.870636 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:58.870651 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:58.870675 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:58.871416 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:58.871496 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:58.871517 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:58.871615 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:58.871925 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 11:29:58.872332 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":410,"threads_started":1}
I20260807 11:29:58.873318 31193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa: No bootstrap required, opened a new log
I20260807 11:29:58.877733 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:29:58.883682 31193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13745572500c4cabaa3e2b02136d36fa" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41639 } }
I20260807 11:29:58.884106 31193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:58.884182 31193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13745572500c4cabaa3e2b02136d36fa, State: Initialized, Role: FOLLOWER
I20260807 11:29:58.884847 31193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13745572500c4cabaa3e2b02136d36fa" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41639 } }
I20260807 11:29:58.885133 31193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:58.885318 31193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:58.885530 31193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:58.888252 31193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13745572500c4cabaa3e2b02136d36fa" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41639 } }
I20260807 11:29:58.888854 31193 leader_election.cc:302] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13745572500c4cabaa3e2b02136d36fa; no voters:
I20260807 11:29:58.889750 31193 leader_election.cc:288] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:58.889794 31199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:58.890558 31199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [term 1 LEADER]: Becoming Leader. State: Replica: 13745572500c4cabaa3e2b02136d36fa, State: Running, Role: LEADER
I20260807 11:29:58.891271 31199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13745572500c4cabaa3e2b02136d36fa" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41639 } }
I20260807 11:29:58.892127 31193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:58.894033 31200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13745572500c4cabaa3e2b02136d36fa. Latest consensus state: current_term: 1 leader_uuid: "13745572500c4cabaa3e2b02136d36fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13745572500c4cabaa3e2b02136d36fa" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41639 } } }
I20260807 11:29:58.894256 31201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13745572500c4cabaa3e2b02136d36fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13745572500c4cabaa3e2b02136d36fa" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41639 } } }
I20260807 11:29:58.894524 31200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:58.894539 31201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:58.895561 31204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:58.901216 31204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:58.911955 31204 catalog_manager.cc:1347] Generated new cluster ID: d446bd47400640eaa6db39f9935f762c
I20260807 11:29:58.912195 31204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:58.921963 31204 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:58.923449 31204 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:58.932543 31204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13745572500c4cabaa3e2b02136d36fa: Generated new TSK 0
I20260807 11:29:58.933497 31204 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 11:29:58.967609 31224 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 11:29:58.967751 31224 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 11:29:58.969049 31196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:58.949394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.24.103.126:41639)
0807 11:29:58.949602 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:58.949608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:58.949982 (+ 374us) server_negotiation.cc:407] Connection header received
0807 11:29:58.950203 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:58.950209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:58.950307 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:58.950491 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0807 11:29:58.950968 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.951827 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.953241 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.953751 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.953903 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:58.954439 (+ 536us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 11:29:58.968253 (+ 13814us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 11:29:58.968674 (+ 421us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":83}
W20260807 11:29:58.969193 31196 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260807 11:29:58.969368 31223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:29:58.949246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41639 (local address 127.0.0.1:35574)
0807 11:29:58.949870 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:58.949890 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:58.950014 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:58.950526 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:58.950531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:58.950540 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0807 11:29:58.950778 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:58.950785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.951981 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.951988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:58.953082 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:58.953094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:58.953966 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:58.953976 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:58.954137 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:58.954172 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 11:29:58.968522 (+ 14350us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 11:29:58.968902 (+ 380us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.103.126:41639: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":343,"threads_started":1}
I20260807 11:29:58.970672 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 31124
I20260807 11:29:58.971957 31188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:29:59.066125 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 31124
2026-08-07T11:29:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (782 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 11:29:59.125293 24989 mini_oidc.cc:152] Starting JWKS server
I20260807 11:29:59.127274 24989 webserver.cc:460] Webserver started at https://127.0.0.1:45903/ using document root <none> and password file <none>
I20260807 11:29:59.127475 24989 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 11:29:59.128070 24989 webserver.cc:460] Webserver started at http://127.0.0.1:43527/ using document root <none> and password file <none>
2026-08-07T11:29:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:59Z Disabled control of system clock
I20260807 11:29:59.162636 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39001
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:39001
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45903/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 11:29:59.504729 31232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:29:59.505034 31232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:29:59.505210 31232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:29:59.512605 31232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:29:59.512718 31232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:29:59.512792 31232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:29:59.512882 31232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:29:59.512992 31232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 11:29:59.513115 31232 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 11:29:59.513175 31232 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45903/jwks.json
I20260807 11:29:59.522353 31232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:39001
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45903/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:29:59.523825 31232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:29:59.525647 31232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:29:59.537878 31237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:59.537878 31238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:29:59.537910 31240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:29:59.538992 31232 server_base.cc:1034] running on GCE node
I20260807 11:29:59.539846 31232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:29:59.544866 31232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:29:59.546113 31232 hybrid_clock.cc:648] HybridClock initialized: now 1786102199546065 us; error 44 us; skew 500 ppm
I20260807 11:29:59.546528 31232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:29:59.552239 31232 webserver.cc:460] Webserver started at http://127.24.103.126:41435/ using document root <none> and password file <none>
I20260807 11:29:59.552977 31232 fs_manager.cc:359] Metadata directory not provided
I20260807 11:29:59.553120 31232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:29:59.553481 31232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:29:59.556630 31232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "32dc4ce807ff4fef9dca7d6eeb339a6f"
format_stamp: "Formatted at 2026-08-07 11:29:59 on dist-test-slave-wl2h"
server_key: "537da18a726ff4e435e637ddec3a8dde"
server_key_iv: "a72a956c6a6a1712f629336ee8adb693"
server_key_version: "encryptionkey@0"
I20260807 11:29:59.557458 31232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "32dc4ce807ff4fef9dca7d6eeb339a6f"
format_stamp: "Formatted at 2026-08-07 11:29:59 on dist-test-slave-wl2h"
server_key: "537da18a726ff4e435e637ddec3a8dde"
server_key_iv: "a72a956c6a6a1712f629336ee8adb693"
server_key_version: "encryptionkey@0"
I20260807 11:29:59.562840 31232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260807 11:29:59.566351 31246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:29:59.567683 31232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:29:59.567871 31232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "32dc4ce807ff4fef9dca7d6eeb339a6f"
format_stamp: "Formatted at 2026-08-07 11:29:59 on dist-test-slave-wl2h"
server_key: "537da18a726ff4e435e637ddec3a8dde"
server_key_iv: "a72a956c6a6a1712f629336ee8adb693"
server_key_version: "encryptionkey@0"
I20260807 11:29:59.568116 31232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:29:59.585721 31232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:29:59.586530 31232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:29:59.586841 31232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:29:59.606189 31232 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:39001
I20260807 11:29:59.606208 31304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:39001 every 8 connection(s)
I20260807 11:29:59.607956 31232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:29:59.611821 31305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:29:59.616081 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 31232
I20260807 11:29:59.616333 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:29:59.617187 24989 external_mini_cluster.cc:1428] Setting key 79578ba05845dece1fcc1df7c610a7f4
I20260807 11:29:59.623593 31305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f: Bootstrap starting.
I20260807 11:29:59.626750 31305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f: Neither blocks nor log segments found. Creating new log.
I20260807 11:29:59.627830 31305 log.cc:824] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f: Log is configured to *not* fsync() on all Append() calls
I20260807 11:29:59.630630 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:59.619419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.24.103.126:39001)
0807 11:29:59.620458 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:59.620468 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:59.620509 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:29:59.622259 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:59.622279 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:59.622680 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:59.623060 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:29:59.623589 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:59.624905 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:59.626245 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:59.626979 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:59.627267 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:59.628409 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:29:59.628420 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:29:59.628432 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:29:59.628955 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:29:59.629060 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:29:59.629087 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:29:59.629220 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 11:29:59.629425 (+ 205us) server_negotiation.cc:299] Negotiation successful
0807 11:29:59.629731 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":369,"threads_started":1}
I20260807 11:29:59.630854 31305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f: No bootstrap required, opened a new log
I20260807 11:29:59.634588 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:29:59.639074 31305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc4ce807ff4fef9dca7d6eeb339a6f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39001 } }
I20260807 11:29:59.639447 31305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:29:59.639567 31305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32dc4ce807ff4fef9dca7d6eeb339a6f, State: Initialized, Role: FOLLOWER
I20260807 11:29:59.640170 31305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc4ce807ff4fef9dca7d6eeb339a6f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39001 } }
I20260807 11:29:59.640410 31305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:29:59.640561 31305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:29:59.640733 31305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:29:59.643234 31305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc4ce807ff4fef9dca7d6eeb339a6f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39001 } }
I20260807 11:29:59.643761 31305 leader_election.cc:302] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32dc4ce807ff4fef9dca7d6eeb339a6f; no voters:
I20260807 11:29:59.644383 31305 leader_election.cc:288] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:29:59.644551 31310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:29:59.645938 31310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [term 1 LEADER]: Becoming Leader. State: Replica: 32dc4ce807ff4fef9dca7d6eeb339a6f, State: Running, Role: LEADER
I20260807 11:29:59.646524 31310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc4ce807ff4fef9dca7d6eeb339a6f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39001 } }
I20260807 11:29:59.646899 31305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:29:59.649027 31311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32dc4ce807ff4fef9dca7d6eeb339a6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc4ce807ff4fef9dca7d6eeb339a6f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39001 } } }
I20260807 11:29:59.649123 31312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32dc4ce807ff4fef9dca7d6eeb339a6f. Latest consensus state: current_term: 1 leader_uuid: "32dc4ce807ff4fef9dca7d6eeb339a6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc4ce807ff4fef9dca7d6eeb339a6f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39001 } } }
I20260807 11:29:59.649547 31312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:59.649554 31311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f [sys.catalog]: This master's current role is: LEADER
I20260807 11:29:59.651190 31316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:29:59.656213 31316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:29:59.665741 31316 catalog_manager.cc:1347] Generated new cluster ID: d541cc198b0c48cf94e08f1fb9c57495
I20260807 11:29:59.665910 31316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:29:59.697213 31316 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:29:59.698714 31316 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:29:59.712102 31316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32dc4ce807ff4fef9dca7d6eeb339a6f: Generated new TSK 0
I20260807 11:29:59.712970 31316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:29:59.742501 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:29:59.733733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.24.103.126:39001)
0807 11:29:59.733994 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 11:29:59.734001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:29:59.734397 (+ 396us) server_negotiation.cc:407] Connection header received
0807 11:29:59.734710 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:29:59.734714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:29:59.734798 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:29:59.734932 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0807 11:29:59.735537 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:59.736295 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:59.738397 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:29:59.738884 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:29:59.738994 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:59.739685 (+ 691us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 11:29:59.741788 (+ 2103us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45903/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 11:29:59.742164 (+ 376us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60570: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45903/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":81}
W20260807 11:29:59.742659 31308 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60570: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45903/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260807 11:29:59.742771 31336 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:29:59.733619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39001 (local address 127.0.0.1:60570)
0807 11:29:59.734274 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0807 11:29:59.734295 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:29:59.734464 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:29:59.734978 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:29:59.734984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:29:59.734998 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0807 11:29:59.735370 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:59.735386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:59.736447 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:59.736453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:59.738201 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:29:59.738217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:29:59.739136 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:29:59.739144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:29:59.739353 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:29:59.739400 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 11:29:59.742025 (+ 2625us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45903/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 11:29:59.742289 (+ 264us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.103.126:39001: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45903/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":332,"threads_started":1}
I20260807 11:29:59.744426 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 31232
I20260807 11:29:59.745689 31299 generic_service.cc:196] Checking for leaks (request via RPC)
E20260807 11:29:59.760180 31337 webserver.cc:533] Webserver: sslize failed:
E20260807 11:29:59.760341 31337 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 11:29:59.844774 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 31232
2026-08-07T11:29:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (779 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T11:29:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:29:59Z Disabled control of system clock
I20260807 11:29:59.939474 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41241
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:41241
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/testchainca.cert with env {}
W20260807 11:30:00.280735 31342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:00.281049 31342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:00.281137 31342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:00.288628 31342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:00.288749 31342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:00.288789 31342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:00.288838 31342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:00.288882 31342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 11:30:00.288945 31342 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 11:30:00.288985 31342 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 11:30:00.299170 31342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:41241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:00.300493 31342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:00.301741 31342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:00.314306 31347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:00.314404 31350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:00.314365 31348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:00.315606 31342 server_base.cc:1034] running on GCE node
I20260807 11:30:00.316515 31342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:00.318065 31342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:00.319309 31342 hybrid_clock.cc:648] HybridClock initialized: now 1786102200319267 us; error 42 us; skew 500 ppm
I20260807 11:30:00.319698 31342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:30:00.322243 31342 webserver.cc:460] Webserver started at http://127.24.103.126:37515/ using document root <none> and password file <none>
I20260807 11:30:00.322997 31342 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:00.323235 31342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:00.323602 31342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:00.326694 31342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "67d16610e3eb4e25a73c813667bcc394"
format_stamp: "Formatted at 2026-08-07 11:30:00 on dist-test-slave-wl2h"
server_key: "db0f5163bb7445832ef2fd4fdd728d42"
server_key_iv: "a21b588430908f6a4e40af9ba9b4317a"
server_key_version: "encryptionkey@0"
I20260807 11:30:00.327566 31342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "67d16610e3eb4e25a73c813667bcc394"
format_stamp: "Formatted at 2026-08-07 11:30:00 on dist-test-slave-wl2h"
server_key: "db0f5163bb7445832ef2fd4fdd728d42"
server_key_iv: "a21b588430908f6a4e40af9ba9b4317a"
server_key_version: "encryptionkey@0"
I20260807 11:30:00.332849 31342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:30:00.336545 31357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:00.337960 31342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:30:00.338169 31342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "67d16610e3eb4e25a73c813667bcc394"
format_stamp: "Formatted at 2026-08-07 11:30:00 on dist-test-slave-wl2h"
server_key: "db0f5163bb7445832ef2fd4fdd728d42"
server_key_iv: "a21b588430908f6a4e40af9ba9b4317a"
server_key_version: "encryptionkey@0"
I20260807 11:30:00.338423 31342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:00.361491 31342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:00.362437 31342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:00.362776 31342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:00.383574 31342 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:41241
I20260807 11:30:00.383594 31414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:41241 every 8 connection(s)
I20260807 11:30:00.385018 31342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:30:00.385797 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 31342
I20260807 11:30:00.386070 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:30:00.386973 24989 external_mini_cluster.cc:1428] Setting key f1257b49915e6fa904d8d765f758a768
I20260807 11:30:00.389532 31415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:00.401347 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:00.389100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.24.103.126:41241)
0807 11:30:00.390109 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:00.390123 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:00.390162 (+ 39us) server_negotiation.cc:407] Connection header received
0807 11:30:00.391861 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:00.391881 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:00.392341 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:00.392688 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:00.393319 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:00.394647 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:00.396562 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:00.397234 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:00.397371 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:00.398775 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:00.398785 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:00.398799 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:00.399460 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:00.399548 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:00.399570 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:00.399673 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:00.399927 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 11:30:00.400337 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":386,"threads_started":1}
I20260807 11:30:00.402743 31415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394: Bootstrap starting.
I20260807 11:30:00.405448 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:30:00.406204 31415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:00.407262 31415 log.cc:824] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:00.410343 31415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394: No bootstrap required, opened a new log
I20260807 11:30:00.417943 31415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d16610e3eb4e25a73c813667bcc394" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41241 } }
I20260807 11:30:00.418344 31415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:00.418453 31415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d16610e3eb4e25a73c813667bcc394, State: Initialized, Role: FOLLOWER
I20260807 11:30:00.418993 31415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d16610e3eb4e25a73c813667bcc394" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41241 } }
I20260807 11:30:00.419299 31415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:30:00.419446 31415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:30:00.419631 31415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:00.421921 31415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d16610e3eb4e25a73c813667bcc394" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41241 } }
I20260807 11:30:00.422374 31415 leader_election.cc:302] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67d16610e3eb4e25a73c813667bcc394; no voters:
I20260807 11:30:00.422959 31415 leader_election.cc:288] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:30:00.423156 31420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:00.423943 31420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [term 1 LEADER]: Becoming Leader. State: Replica: 67d16610e3eb4e25a73c813667bcc394, State: Running, Role: LEADER
I20260807 11:30:00.424485 31420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d16610e3eb4e25a73c813667bcc394" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41241 } }
I20260807 11:30:00.425268 31415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:00.427605 31422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67d16610e3eb4e25a73c813667bcc394. Latest consensus state: current_term: 1 leader_uuid: "67d16610e3eb4e25a73c813667bcc394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d16610e3eb4e25a73c813667bcc394" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41241 } } }
I20260807 11:30:00.427907 31422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:00.429266 31421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67d16610e3eb4e25a73c813667bcc394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d16610e3eb4e25a73c813667bcc394" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41241 } } }
I20260807 11:30:00.429718 31421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:00.429944 31427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:30:00.435330 31427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:30:00.445106 31427 catalog_manager.cc:1347] Generated new cluster ID: b43c50ad604f429d9cf757142ffe15c9
I20260807 11:30:00.445246 31427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:30:00.469810 31427 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:30:00.471683 31427 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:30:00.480129 31427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67d16610e3eb4e25a73c813667bcc394: Generated new TSK 0
I20260807 11:30:00.481194 31427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:30:00.958079 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:00.498051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.24.103.126:41241)
0807 11:30:00.498280 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:00.498286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:00.498700 (+ 414us) server_negotiation.cc:407] Connection header received
0807 11:30:00.498927 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:00.498934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:00.499069 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:00.499272 (+ 203us) server_negotiation.cc:226] Negotiated authn=JWT
0807 11:30:00.499697 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:00.500862 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:00.502326 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:00.503052 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:00.503177 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:00.503749 (+ 572us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 11:30:00.957450 (+453701us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 11:30:00.957841 (+ 391us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42992: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 11:30:00.958329 31418 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42992: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260807 11:30:00.958488 31444 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:00.497857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41241 (local address 127.0.0.1:42992)
0807 11:30:00.498591 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:00.498608 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:00.498734 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:00.499284 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:00.499287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:00.499298 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0807 11:30:00.499537 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:00.499547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:00.501042 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:00.501046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:00.502171 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:00.502184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:00.503238 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:00.503243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:00.503386 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:00.503423 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 11:30:00.957804 (+454381us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 11:30:00.958078 (+ 274us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.103.126:41241: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":436,"threads_started":1}
I20260807 11:30:00.959853 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 31342
I20260807 11:30:00.961092 31410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:01.064718 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 31342
2026-08-07T11:30:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1202 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (433 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1598 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:30:03 dist-test-slave-wl2h krb5kdc[31463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:30:03 dist-test-slave-wl2h krb5kdc[31463](info): set up 2 sockets
Aug 07 11:30:03 dist-test-slave-wl2h krb5kdc[31463](info): commencing operation
krb5kdc: starting...
W20260807 11:30:04.307183 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:30:04 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:30:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:30:04.579500 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39185
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:04.918892 31481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:04.919281 31481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:04.919445 31481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:04.926656 31481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:04.926805 31481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:04.926924 31481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:04.927050 31481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:04.927134 31481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:30:04.937175 31481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:39185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:04.938591 31481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:04.939946 31481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:04.952538 31490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:04.952637 31488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:04.953186 31487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:04.953537 31481 server_base.cc:1034] running on GCE node
I20260807 11:30:04.954363 31481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:04.955981 31481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:04.957280 31481 hybrid_clock.cc:648] HybridClock initialized: now 1786102204957209 us; error 39 us; skew 500 ppm
I20260807 11:30:04.957734 31481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:04 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102204, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:04.966784 31481 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:30:04.968689 31481 webserver.cc:460] Webserver started at http://127.24.103.126:44523/ using document root <none> and password file <none>
I20260807 11:30:04.969441 31481 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:04.969588 31481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:04.969992 31481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:04.973237 31481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "70e51368ee42439fbb5a8ec5bb4ab429"
format_stamp: "Formatted at 2026-08-07 11:30:04 on dist-test-slave-wl2h"
server_key: "82f44e8ba486f1fe199a1ee0d1d05abe"
server_key_iv: "fd021e16271f24b5a111937bd6c09321"
server_key_version: "encryptionkey@0"
I20260807 11:30:04.974064 31481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "70e51368ee42439fbb5a8ec5bb4ab429"
format_stamp: "Formatted at 2026-08-07 11:30:04 on dist-test-slave-wl2h"
server_key: "82f44e8ba486f1fe199a1ee0d1d05abe"
server_key_iv: "fd021e16271f24b5a111937bd6c09321"
server_key_version: "encryptionkey@0"
I20260807 11:30:04.979751 31481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260807 11:30:04.983704 31497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:04.985476 31481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260807 11:30:04.985683 31481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "70e51368ee42439fbb5a8ec5bb4ab429"
format_stamp: "Formatted at 2026-08-07 11:30:04 on dist-test-slave-wl2h"
server_key: "82f44e8ba486f1fe199a1ee0d1d05abe"
server_key_iv: "fd021e16271f24b5a111937bd6c09321"
server_key_version: "encryptionkey@0"
I20260807 11:30:04.985890 31481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:05.006523 31481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:05.013165 31481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:05.013855 31481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:05.034844 31481 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:39185
I20260807 11:30:05.034946 31554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:39185 every 8 connection(s)
I20260807 11:30:05.036429 31481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:30:05.040261 31555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:05.046626 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 31481
I20260807 11:30:05.046864 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:30:05.047607 24989 external_mini_cluster.cc:1428] Setting key a8de64a18eacdbd433b034cafbfa7094
I20260807 11:30:05.050724 31555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429: Bootstrap starting.
I20260807 11:30:05.054261 31555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:05.055501 31555 log.cc:824] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:05.058807 31555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429: No bootstrap required, opened a new log
Aug 07 11:30:05 dist-test-slave-wl2h krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:05.067945 31555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e51368ee42439fbb5a8ec5bb4ab429" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39185 } }
I20260807 11:30:05.068372 31555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:05.068481 31555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e51368ee42439fbb5a8ec5bb4ab429, State: Initialized, Role: FOLLOWER
I20260807 11:30:05.069155 31555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e51368ee42439fbb5a8ec5bb4ab429" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39185 } }
I20260807 11:30:05.069419 31555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:30:05.069602 31555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:30:05.069799 31555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:05.072400 31555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e51368ee42439fbb5a8ec5bb4ab429" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39185 } }
I20260807 11:30:05.072476 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:05.049598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.24.103.126:39185)
0807 11:30:05.050375 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:05.050387 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:05.050429 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:30:05.052335 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:05.052356 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:05.052733 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:05.053039 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:05.054902 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:05.056610 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:05.058548 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:05.059377 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:05.059503 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:05.064464 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:05.064473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:05.064489 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:05.064544 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:05.067910 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:05.068807 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:05.068815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:05.068821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:05.069001 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:05.069422 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:05.069428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:05.069431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:05.070146 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:05.070467 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:05.070940 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 11:30:05.071432 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":372,"threads_started":1}
I20260807 11:30:05.073021 31555 leader_election.cc:302] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e51368ee42439fbb5a8ec5bb4ab429; no voters:
I20260807 11:30:05.073755 31555 leader_election.cc:288] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:30:05.073920 31560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:05.075838 31555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:05.076403 31560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [term 1 LEADER]: Becoming Leader. State: Replica: 70e51368ee42439fbb5a8ec5bb4ab429, State: Running, Role: LEADER
I20260807 11:30:05.077039 31560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e51368ee42439fbb5a8ec5bb4ab429" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39185 } }
I20260807 11:30:05.080106 31562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e51368ee42439fbb5a8ec5bb4ab429. Latest consensus state: current_term: 1 leader_uuid: "70e51368ee42439fbb5a8ec5bb4ab429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e51368ee42439fbb5a8ec5bb4ab429" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39185 } } }
I20260807 11:30:05.080472 31562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:05.080901 31561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70e51368ee42439fbb5a8ec5bb4ab429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e51368ee42439fbb5a8ec5bb4ab429" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39185 } } }
I20260807 11:30:05.081202 31561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:05.081444 31574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:30:05.087611 31574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:30:05.098771 31574 catalog_manager.cc:1347] Generated new cluster ID: a71f01904691414baa8cdc6ab175330a
I20260807 11:30:05.098913 31574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:30:05.133930 31574 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:30:05.135897 31574 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:30:05.151454 31574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70e51368ee42439fbb5a8ec5bb4ab429: Generated new TSK 0
I20260807 11:30:05.152554 31574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:30:05.228044 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39185
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:05.587677 31585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:05.588018 31585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:05.588186 31585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:05.596015 31585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:05.596182 31585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:05.596360 31585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:30:05.606535 31585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:05.608022 31585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:05.609447 31585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:05.623386 31592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:05.623464 31594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:05.623486 31591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:05.624488 31585 server_base.cc:1034] running on GCE node
I20260807 11:30:05.625123 31585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:05.626101 31585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:05.627364 31585 hybrid_clock.cc:648] HybridClock initialized: now 1786102205627299 us; error 58 us; skew 500 ppm
I20260807 11:30:05.627712 31585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:05 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102205, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:05.631892 31585 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:30:05.633385 31585 webserver.cc:460] Webserver started at http://127.24.103.65:41577/ using document root <none> and password file <none>
I20260807 11:30:05.634047 31585 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:05.634147 31585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:05.634390 31585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:05.637437 31585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "327f7f4316b347d38741c387ba6e93a7"
format_stamp: "Formatted at 2026-08-07 11:30:05 on dist-test-slave-wl2h"
server_key: "2b6a8c94d48affe47a8decdf52d918f0"
server_key_iv: "136076e397cdbab4370abfe2cdd4e12e"
server_key_version: "encryptionkey@0"
I20260807 11:30:05.638224 31585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "327f7f4316b347d38741c387ba6e93a7"
format_stamp: "Formatted at 2026-08-07 11:30:05 on dist-test-slave-wl2h"
server_key: "2b6a8c94d48affe47a8decdf52d918f0"
server_key_iv: "136076e397cdbab4370abfe2cdd4e12e"
server_key_version: "encryptionkey@0"
I20260807 11:30:05.643822 31585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 11:30:05.647521 31602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:05.649015 31585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:30:05.649221 31585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "327f7f4316b347d38741c387ba6e93a7"
format_stamp: "Formatted at 2026-08-07 11:30:05 on dist-test-slave-wl2h"
server_key: "2b6a8c94d48affe47a8decdf52d918f0"
server_key_iv: "136076e397cdbab4370abfe2cdd4e12e"
server_key_version: "encryptionkey@0"
I20260807 11:30:05.649438 31585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:05.676343 31585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:05.682016 31585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:05.682420 31585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:05.683595 31585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:05.685117 31585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:05.685247 31585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:05.685398 31585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:05.685475 31585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:05.722651 31585 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:37321
I20260807 11:30:05.722715 31731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:37321 every 8 connection(s)
I20260807 11:30:05.724007 31585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:30:05.727303 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 31585
I20260807 11:30:05.727530 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:30:05.728304 24989 external_mini_cluster.cc:1428] Setting key 0140a6befea0d5ce50a7c6f578f332da
Aug 07 11:30:05 dist-test-slave-wl2h krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102205, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:05.749048 31735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:05.726999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:46783 (local address 127.24.103.126:39185)
0807 11:30:05.727655 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:05.727663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:05.728811 (+ 1148us) server_negotiation.cc:407] Connection header received
0807 11:30:05.731205 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:05.731209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:05.731281 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:05.731381 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:05.733766 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:05.734536 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:05.737161 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:05.737605 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:05.737730 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:05.742405 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:05.742411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:05.742414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:05.742452 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:05.745067 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:05.746176 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:05.746180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:05.746183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:05.746314 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:05.746897 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:05.746901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:05.746903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:05.747337 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:05.747500 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:05.748568 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0807 11:30:05.748759 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":296,"threads_started":1}
I20260807 11:30:05.750631 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:05.727642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39185 (local address 127.24.103.65:46783)
0807 11:30:05.728591 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:05.728643 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:05.730825 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:05.731695 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:05.731716 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:05.732644 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:05.733546 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:05.733578 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:05.734732 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:05.734740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:05.736973 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:05.736994 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:05.737891 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:05.737900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:05.738261 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:05.739186 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:05.739232 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:05.742167 (+ 2935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:05.745233 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:05.745246 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:05.745268 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:05.745987 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:05.746447 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:05.746455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:05.746461 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:05.746736 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:05.747493 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:05.747504 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:05.748050 (+ 546us) client_negotiation.cc:769] Sending connection context
0807 11:30:05.748400 (+ 350us) client_negotiation.cc:240] Negotiation successful
0807 11:30:05.749041 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":351,"threads_started":1}
I20260807 11:30:05.752393 31732 heartbeater.cc:347] Connected to a master server at 127.24.103.126:39185
I20260807 11:30:05.752756 31732 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:05.753616 31732 heartbeater.cc:511] Master 127.24.103.126:39185 requested a full tablet report, sending...
I20260807 11:30:05.755942 31517 ts_manager.cc:194] Registered new tserver with Master: 327f7f4316b347d38741c387ba6e93a7 (127.24.103.65:37321)
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260807 11:30:05.757460 31517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:46783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:30:05.791648 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39185
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:06.132503 31739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:06.132807 31739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:06.132972 31739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:06.140252 31739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:06.140368 31739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:06.140551 31739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:30:06.150390 31739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:06.151784 31739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:06.153175 31739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:06.166987 31745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:06.166950 31747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:06.167038 31744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:06.168946 31739 server_base.cc:1034] running on GCE node
I20260807 11:30:06.169901 31739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:06.171243 31739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:06.172547 31739 hybrid_clock.cc:648] HybridClock initialized: now 1786102206172504 us; error 55 us; skew 500 ppm
I20260807 11:30:06.172945 31739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:06.176868 31739 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:30:06.178577 31739 webserver.cc:460] Webserver started at http://127.24.103.66:34845/ using document root <none> and password file <none>
I20260807 11:30:06.179512 31739 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:06.179643 31739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:06.180049 31739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:06.183305 31739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "079a871e391d4a8bb38d7c2c6d989566"
format_stamp: "Formatted at 2026-08-07 11:30:06 on dist-test-slave-wl2h"
server_key: "6b734337c8fa1816f1a249e5285f80de"
server_key_iv: "9adcf6157df5b1d73b71b0de1660157d"
server_key_version: "encryptionkey@0"
I20260807 11:30:06.184137 31739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "079a871e391d4a8bb38d7c2c6d989566"
format_stamp: "Formatted at 2026-08-07 11:30:06 on dist-test-slave-wl2h"
server_key: "6b734337c8fa1816f1a249e5285f80de"
server_key_iv: "9adcf6157df5b1d73b71b0de1660157d"
server_key_version: "encryptionkey@0"
I20260807 11:30:06.190172 31739 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260807 11:30:06.194232 31754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:06.195712 31739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:30:06.195904 31739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "079a871e391d4a8bb38d7c2c6d989566"
format_stamp: "Formatted at 2026-08-07 11:30:06 on dist-test-slave-wl2h"
server_key: "6b734337c8fa1816f1a249e5285f80de"
server_key_iv: "9adcf6157df5b1d73b71b0de1660157d"
server_key_version: "encryptionkey@0"
I20260807 11:30:06.196100 31739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:06.218675 31739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:06.223557 31739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:06.223872 31739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:06.224922 31739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:06.226480 31739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:06.226557 31739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:06.226696 31739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:06.226775 31739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:06.263556 31739 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:37641
I20260807 11:30:06.263615 31878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:37641 every 8 connection(s)
I20260807 11:30:06.264982 31739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:30:06.269649 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 31739
I20260807 11:30:06.269850 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:30:06.270447 24989 external_mini_cluster.cc:1428] Setting key 4159691de2d0323cdb8863cf0275aaf4
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:06.292984 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:06.267909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:60291 (local address 127.24.103.126:39185)
0807 11:30:06.268427 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:06.268432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:06.269712 (+ 1280us) server_negotiation.cc:407] Connection header received
0807 11:30:06.272176 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:06.272183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:06.272260 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:06.272400 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:06.274919 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.276037 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.279427 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.279966 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.280092 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:06.285358 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:06.285368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:06.285375 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:06.285426 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:06.289387 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.290251 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.290259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.290266 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.290431 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.290984 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.290988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.290990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.291361 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:06.291539 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:06.292339 (+ 800us) server_negotiation.cc:299] Negotiation successful
0807 11:30:06.292577 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":272,"threads_started":1}
I20260807 11:30:06.294025 31882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:06.268499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39185 (local address 127.24.103.66:60291)
0807 11:30:06.269486 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:06.269550 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:06.271864 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:06.272854 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:06.272864 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:06.273726 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:06.274665 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:06.274700 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.276233 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.276242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:06.278984 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:06.279006 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.280286 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.280298 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:06.280677 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:06.281586 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:06.281624 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:06.285016 (+ 3392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:06.289574 (+ 4558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:06.289582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:06.289597 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:06.290104 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:06.290601 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:06.290606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:06.290611 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:06.290836 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:06.291509 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:06.291517 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:06.291903 (+ 386us) client_negotiation.cc:769] Sending connection context
0807 11:30:06.292223 (+ 320us) client_negotiation.cc:240] Negotiation successful
0807 11:30:06.292801 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":394,"threads_started":1}
I20260807 11:30:06.295681 31879 heartbeater.cc:347] Connected to a master server at 127.24.103.126:39185
I20260807 11:30:06.296057 31879 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:06.297070 31879 heartbeater.cc:511] Master 127.24.103.126:39185 requested a full tablet report, sending...
I20260807 11:30:06.298969 31517 ts_manager.cc:194] Registered new tserver with Master: 079a871e391d4a8bb38d7c2c6d989566 (127.24.103.66:37641)
I20260807 11:30:06.300359 31517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:60291
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:30:06.339792 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39185
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:06.699378 31885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:06.699736 31885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:06.699916 31885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:06.708516 31885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:06.708698 31885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:06.708905 31885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:30:06.720213 31885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:06.721987 31885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:06.723536 31885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:06.738392 31890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:06.738314 31893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:06.738291 31891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:06.738904 31885 server_base.cc:1034] running on GCE node
I20260807 11:30:06.739647 31885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:06.740693 31885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:06.741946 31885 hybrid_clock.cc:648] HybridClock initialized: now 1786102206741880 us; error 57 us; skew 500 ppm
I20260807 11:30:06.742308 31885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:06.745779 31885 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:30:06.747509 31885 webserver.cc:460] Webserver started at http://127.24.103.67:41327/ using document root <none> and password file <none>
I20260807 11:30:06.748231 31885 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:06.748386 31885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:06.748728 31885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:06.752008 31885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "ed49d0ab23664ea2b98d84534720a089"
format_stamp: "Formatted at 2026-08-07 11:30:06 on dist-test-slave-wl2h"
server_key: "fa36fb1ee79d3f0bd1febf65b58f9eb0"
server_key_iv: "dca939493edf3d3dd3320d46a7300959"
server_key_version: "encryptionkey@0"
I20260807 11:30:06.752956 31885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "ed49d0ab23664ea2b98d84534720a089"
format_stamp: "Formatted at 2026-08-07 11:30:06 on dist-test-slave-wl2h"
server_key: "fa36fb1ee79d3f0bd1febf65b58f9eb0"
server_key_iv: "dca939493edf3d3dd3320d46a7300959"
server_key_version: "encryptionkey@0"
I20260807 11:30:06.759276 31885 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 11:30:06.760769 31732 heartbeater.cc:503] Master 127.24.103.126:39185 was elected leader, sending a full tablet report...
I20260807 11:30:06.763478 31901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:06.764982 31885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:30:06.765129 31885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "ed49d0ab23664ea2b98d84534720a089"
format_stamp: "Formatted at 2026-08-07 11:30:06 on dist-test-slave-wl2h"
server_key: "fa36fb1ee79d3f0bd1febf65b58f9eb0"
server_key_iv: "dca939493edf3d3dd3320d46a7300959"
server_key_version: "encryptionkey@0"
I20260807 11:30:06.765321 31885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:06.778174 31885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:06.783165 31885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:06.783459 31885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:06.784510 31885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:06.785945 31885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:06.786019 31885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:06.786113 31885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:06.786158 31885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:06.823753 31885 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:46395
I20260807 11:30:06.823837 32023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:46395 every 8 connection(s)
I20260807 11:30:06.825194 31885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:30:06.828795 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 31885
I20260807 11:30:06.828995 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:30:06.829474 24989 external_mini_cluster.cc:1428] Setting key d01cd134cdb71521fbd4954f9fa5b49a
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:06.847807 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:06.828126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:59865 (local address 127.24.103.126:39185)
0807 11:30:06.828794 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:06.828801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:06.829932 (+ 1131us) server_negotiation.cc:407] Connection header received
0807 11:30:06.832394 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:06.832409 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:06.832488 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:06.832595 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:06.834385 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.835385 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.837141 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.837539 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.837640 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:06.841425 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:06.841430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:06.841433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:06.841469 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:06.844486 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.845367 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.845374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.845378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.845514 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.845965 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.845972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.845976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.846255 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:06.846425 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:06.847161 (+ 736us) server_negotiation.cc:299] Negotiation successful
0807 11:30:06.847386 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":286,"threads_started":1}
I20260807 11:30:06.848886 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:06.828737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:39185 (local address 127.24.103.67:59865)
0807 11:30:06.829735 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:06.829785 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:06.832070 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:06.832890 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:06.832906 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:06.833609 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:06.834207 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:06.834226 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.835585 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.835590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:06.836982 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:06.837000 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.837759 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.837765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:06.837977 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:06.838723 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:06.838746 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:06.841188 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:06.844650 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:06.844657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:06.844671 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:06.845223 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:06.845634 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:06.845639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:06.845643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:06.845844 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:06.846389 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:06.846398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:06.846760 (+ 362us) client_negotiation.cc:769] Sending connection context
0807 11:30:06.847065 (+ 305us) client_negotiation.cc:240] Negotiation successful
0807 11:30:06.847640 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":482,"threads_started":1}
I20260807 11:30:06.850380 32025 heartbeater.cc:347] Connected to a master server at 127.24.103.126:39185
I20260807 11:30:06.850689 32025 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:06.851667 32025 heartbeater.cc:511] Master 127.24.103.126:39185 requested a full tablet report, sending...
I20260807 11:30:06.853273 31517 ts_manager.cc:194] Registered new tserver with Master: ed49d0ab23664ea2b98d84534720a089 (127.24.103.67:46395)
I20260807 11:30:06.854245 31517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:59865
I20260807 11:30:06.858563 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/test-admin.keytab.
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:06.893929 24989 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 11:30:06.899945 32033 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:06.897075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.24.103.65:37321)
0807 11:30:06.897801 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:06.897812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:06.897853 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:30:06.897915 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:06.897922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:06.898134 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:06.898334 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:06.899424 (+ 1090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54912: BlockingRecv error: recv got EOF from 127.0.0.1:54912 (error 108)
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":352,"threads_started":1}
W20260807 11:30:06.900039 24989 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/krb5cc' of type FILE
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:06.902307 24989 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:06.903959 24989 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:30:06.920730 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:06.906033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.24.103.65:37321)
0807 11:30:06.906265 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:06.906270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:06.906338 (+ 68us) server_negotiation.cc:407] Connection header received
0807 11:30:06.906473 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:06.906477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:06.906538 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:06.906626 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:06.907782 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.908755 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.909992 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.910561 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.910660 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:06.914624 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:06.914636 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:06.914651 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:06.914695 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:06.917958 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.918673 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.918680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.918682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.918805 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.919295 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.919299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.919301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.919692 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:06.919918 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:06.920231 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 11:30:06.920434 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260807 11:30:06.927489 32033 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:06.925356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.24.103.65:37321)
0807 11:30:06.925654 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:06.925662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:06.925972 (+ 310us) server_negotiation.cc:407] Connection header received
0807 11:30:06.926226 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:06.926229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:06.926288 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:06.926392 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:06.927200 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54922: BlockingRecv error: recv got EOF from 127.0.0.1:54922 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
W20260807 11:30:06.927618 24989 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/krb5cc' of type FILE
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:06.929363 24989 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:06.930790 24989 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:30:06.945924 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:06.932255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.24.103.65:37321)
0807 11:30:06.932475 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:06.932481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:06.932763 (+ 282us) server_negotiation.cc:407] Connection header received
0807 11:30:06.932879 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:06.932883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:06.932938 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:06.933017 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:06.934110 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.934882 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.936260 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.936709 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.936849 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:06.940523 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:06.940528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:06.940531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:06.940574 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:06.943350 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.944041 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.944045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.944048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.944185 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.944641 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.944646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.944648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.944924 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:06.945156 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:06.945487 (+ 331us) server_negotiation.cc:299] Negotiation successful
0807 11:30:06.945693 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260807 11:30:06.951550 32033 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:06.949275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.24.103.65:37321)
0807 11:30:06.949516 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:06.949523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:06.949809 (+ 286us) server_negotiation.cc:407] Connection header received
0807 11:30:06.949922 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:06.949925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:06.949997 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:06.950078 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:06.951295 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54940: BlockingRecv error: recv got EOF from 127.0.0.1:54940 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260807 11:30:06.951804 24989 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786102153172159-24989-0/krb5kdc/krb5cc' of type FILE
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:06.953717 24989 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:06.955411 24989 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 11:30:06 dist-test-slave-wl2h krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:30:06.972891 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:06.956986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.24.103.65:37321)
0807 11:30:06.957226 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:06.957231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:06.957703 (+ 472us) server_negotiation.cc:407] Connection header received
0807 11:30:06.957922 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:06.957926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:06.957992 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:06.958096 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:06.959518 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.960321 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.961618 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.962088 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.962207 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:06.966510 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:06.966516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:06.966519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:06.966557 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:06.969478 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.970254 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.970260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.970264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.970496 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.970931 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.970938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.970941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.971482 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:06.971888 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:06.972217 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 11:30:06.972495 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:30:06.975428 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 31585
I20260807 11:30:06.989115 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:06.976258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.24.103.65:37321)
0807 11:30:06.976419 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:06.976423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:06.976996 (+ 573us) server_negotiation.cc:407] Connection header received
0807 11:30:06.977122 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:06.977126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:06.977191 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:06.977299 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:06.978487 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.979465 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.980982 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:06.981466 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:06.981630 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:06.983973 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:06.983978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:06.983982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:06.984019 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:06.986633 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.987390 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.987394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.987396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.987546 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:06.987885 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:06.987888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:06.987890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:06.988166 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:06.988384 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:06.988609 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 11:30:06.988796 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 11:30:06.989769 31727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:07.086388 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 31585
I20260807 11:30:07.107003 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 31739
Aug 07 11:30:07 dist-test-slave-wl2h krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:30:07.124778 32046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:07.108000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.24.103.66:37641)
0807 11:30:07.108566 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:07.108574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:07.108600 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:30:07.108671 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:07.108678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:07.108878 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:07.109015 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:07.110168 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:07.111408 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:07.112905 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:07.113513 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:07.113629 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:07.118100 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:07.118112 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:07.118128 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:07.118166 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:07.121552 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:07.122321 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:07.122328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:07.122330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:07.122459 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:07.122886 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:07.122891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:07.122893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:07.123441 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:07.123774 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:07.124187 (+ 413us) server_negotiation.cc:299] Negotiation successful
0807 11:30:07.124416 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":292,"threads_started":1}
I20260807 11:30:07.126250 31873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:07.213729 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 31739
I20260807 11:30:07.233373 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 31885
Aug 07 11:30:07 dist-test-slave-wl2h krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:30:07.250401 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:07.234545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.24.103.67:46395)
0807 11:30:07.235329 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:07.235337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:07.235369 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:30:07.235431 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:07.235438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:07.235651 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:07.235791 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:07.236856 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:07.237895 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:07.239344 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:07.239910 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:07.240015 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:07.244250 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:07.244263 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:07.244277 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:07.244324 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:07.247347 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:07.248160 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:07.248167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:07.248169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:07.248347 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:07.248820 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:07.248824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:07.248827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:07.249237 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:07.249507 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:07.249842 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 11:30:07.250056 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":401,"threads_started":1}
I20260807 11:30:07.252036 32019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:07.339648 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 31885
I20260807 11:30:07.359272 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 31481
I20260807 11:30:07.360702 31550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:07.469806 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 31481
2026-08-07T11:30:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4377 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:30:07 dist-test-slave-wl2h krb5kdc[32053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:30:07 dist-test-slave-wl2h krb5kdc[32053](info): set up 2 sockets
Aug 07 11:30:07 dist-test-slave-wl2h krb5kdc[32053](info): commencing operation
krb5kdc: starting...
W20260807 11:30:08.696494 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:30:08 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:30:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.126.keytab.
Entry for principal oryx/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.126.keytab.
I20260807 11:30:08.979807 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42143
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.126.keytab
--principal=oryx/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:09.316547 32066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:09.316973 32066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:09.317152 32066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:09.325570 32066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:09.325757 32066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:09.325848 32066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:09.325958 32066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:09.326026 32066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:30:09.336939 32066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.126.keytab
--principal=oryx/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:09.338394 32066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:09.339795 32066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:09.353377 32072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:09.353420 32073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:09.353457 32075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:09.354451 32066 server_base.cc:1034] running on GCE node
I20260807 11:30:09.355439 32066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:09.356954 32066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:09.358211 32066 hybrid_clock.cc:648] HybridClock initialized: now 1786102209358178 us; error 56 us; skew 500 ppm
I20260807 11:30:09.358578 32066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:09 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102209, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:09.361950 32066 init.cc:377] Logged in from keytab as oryx/127.24.103.126@KRBTEST.COM (short username oryx)
I20260807 11:30:09.363690 32066 webserver.cc:460] Webserver started at http://127.24.103.126:36527/ using document root <none> and password file <none>
I20260807 11:30:09.364401 32066 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:09.364508 32066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:09.364784 32066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:09.368221 32066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "7182993a35ce4f5aa488a3caa7acea01"
format_stamp: "Formatted at 2026-08-07 11:30:09 on dist-test-slave-wl2h"
server_key: "ec7489b3532537e6bdfd99b3f85a6815"
server_key_iv: "fae7306bc990a226d0ae29d3fa90103e"
server_key_version: "encryptionkey@0"
I20260807 11:30:09.368980 32066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "7182993a35ce4f5aa488a3caa7acea01"
format_stamp: "Formatted at 2026-08-07 11:30:09 on dist-test-slave-wl2h"
server_key: "ec7489b3532537e6bdfd99b3f85a6815"
server_key_iv: "fae7306bc990a226d0ae29d3fa90103e"
server_key_version: "encryptionkey@0"
I20260807 11:30:09.374661 32066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:30:09.380048 32083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:09.381946 32066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:30:09.382161 32066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "7182993a35ce4f5aa488a3caa7acea01"
format_stamp: "Formatted at 2026-08-07 11:30:09 on dist-test-slave-wl2h"
server_key: "ec7489b3532537e6bdfd99b3f85a6815"
server_key_iv: "fae7306bc990a226d0ae29d3fa90103e"
server_key_version: "encryptionkey@0"
I20260807 11:30:09.382455 32066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:09.419553 32066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:09.424347 32066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:09.424620 32066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:09.444011 32066 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:42143
I20260807 11:30:09.444046 32141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:42143 every 8 connection(s)
I20260807 11:30:09.445533 32066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:30:09.447424 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 32066
I20260807 11:30:09.447664 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:30:09.448721 24989 external_mini_cluster.cc:1428] Setting key c65ea399790f1dcc97d7b399d270423f
I20260807 11:30:09.450538 32142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:30:09 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:09.466522 32142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01: Bootstrap starting.
I20260807 11:30:09.469959 32142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:09.471405 32142 log.cc:824] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:09.474709 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:09.451081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.24.103.126:42143)
0807 11:30:09.451949 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:09.451960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:09.452002 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:30:09.453706 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:09.453728 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:09.454129 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:09.454483 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:09.456746 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:09.458421 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:09.460268 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:09.460985 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:09.461144 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:09.466440 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:09.466450 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:09.466467 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:09.466517 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:09.469828 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:09.470834 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:09.470842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:09.470849 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:09.471116 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:09.471624 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:09.471630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:09.471633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:09.472478 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:09.472842 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:09.473302 (+ 460us) server_negotiation.cc:299] Negotiation successful
0807 11:30:09.473632 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":404,"threads_started":1}
I20260807 11:30:09.475222 32142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01: No bootstrap required, opened a new log
I20260807 11:30:09.483686 32142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } }
I20260807 11:30:09.484232 32142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:09.484349 32142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7182993a35ce4f5aa488a3caa7acea01, State: Initialized, Role: FOLLOWER
I20260807 11:30:09.485021 32142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } }
I20260807 11:30:09.485278 32142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:30:09.485437 32142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:30:09.485617 32142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:09.488363 32142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } }
I20260807 11:30:09.488987 32142 leader_election.cc:302] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7182993a35ce4f5aa488a3caa7acea01; no voters:
I20260807 11:30:09.489693 32142 leader_election.cc:288] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:30:09.489862 32148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:09.490576 32148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 1 LEADER]: Becoming Leader. State: Replica: 7182993a35ce4f5aa488a3caa7acea01, State: Running, Role: LEADER
I20260807 11:30:09.491237 32148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } }
I20260807 11:30:09.492007 32142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:09.494179 32151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7182993a35ce4f5aa488a3caa7acea01. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } }
I20260807 11:30:09.494603 32151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:09.495168 32150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } }
I20260807 11:30:09.495497 32150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:09.495868 32159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:30:09.502278 32159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:30:09.511895 32159 catalog_manager.cc:1347] Generated new cluster ID: 13e0d15ed169414d8d7b9cb933658be1
I20260807 11:30:09.512050 32159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:30:09.528491 32159 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:30:09.529865 32159 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:30:09.537310 32159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01: Generated new TSK 0
I20260807 11:30:09.538158 32159 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
I20260807 11:30:09.612881 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42143
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:09.963515 32172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:09.963809 32172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:09.963974 32172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:09.973835 32172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:09.974018 32172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:09.974280 32172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:30:09.987669 32172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:09.989257 32172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:09.990773 32172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:10.005949 32177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:10.006049 32181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:10.006477 32178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:10.006914 32172 server_base.cc:1034] running on GCE node
I20260807 11:30:10.007644 32172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:10.008747 32172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:10.009989 32172 hybrid_clock.cc:648] HybridClock initialized: now 1786102210009938 us; error 44 us; skew 500 ppm
I20260807 11:30:10.010355 32172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:10 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102210, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:10.018218 32172 init.cc:377] Logged in from keytab as oryx/127.24.103.65@KRBTEST.COM (short username oryx)
I20260807 11:30:10.020025 32172 webserver.cc:460] Webserver started at http://127.24.103.65:43573/ using document root <none> and password file <none>
I20260807 11:30:10.020797 32172 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:10.020952 32172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:10.021283 32172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:10.024421 32172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "7422a6ab063542509f75e3098e0dca05"
format_stamp: "Formatted at 2026-08-07 11:30:10 on dist-test-slave-wl2h"
server_key: "ae926a7e318fa1185bcacd1c057078c5"
server_key_iv: "48efe2c0286950adf2badfea89c185be"
server_key_version: "encryptionkey@0"
I20260807 11:30:10.025249 32172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "7422a6ab063542509f75e3098e0dca05"
format_stamp: "Formatted at 2026-08-07 11:30:10 on dist-test-slave-wl2h"
server_key: "ae926a7e318fa1185bcacd1c057078c5"
server_key_iv: "48efe2c0286950adf2badfea89c185be"
server_key_version: "encryptionkey@0"
I20260807 11:30:10.030710 32172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 11:30:10.034502 32188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:10.036238 32172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:30:10.036439 32172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "7422a6ab063542509f75e3098e0dca05"
format_stamp: "Formatted at 2026-08-07 11:30:10 on dist-test-slave-wl2h"
server_key: "ae926a7e318fa1185bcacd1c057078c5"
server_key_iv: "48efe2c0286950adf2badfea89c185be"
server_key_version: "encryptionkey@0"
I20260807 11:30:10.036660 32172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:10.069939 32172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:10.074576 32172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:10.074908 32172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:10.075976 32172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:10.077420 32172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:10.077518 32172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:10.077662 32172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:10.077741 32172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:10.114006 32172 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:46611
I20260807 11:30:10.114099 32312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:46611 every 8 connection(s)
I20260807 11:30:10.115398 32172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:30:10.122294 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 32172
I20260807 11:30:10.122541 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:30:10.123410 24989 external_mini_cluster.cc:1428] Setting key 84b840541ba58b3271e0e7362f5a52ef
Aug 07 11:30:10 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102210, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:10.143682 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:10.118404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:45371 (local address 127.24.103.126:42143)
0807 11:30:10.118912 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:10.118919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:10.120053 (+ 1134us) server_negotiation.cc:407] Connection header received
0807 11:30:10.123104 (+ 3051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:10.123111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:10.123216 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:10.123350 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:10.125496 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:10.126567 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:10.129232 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:10.129942 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:10.130099 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:10.135127 (+ 5028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:10.135136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:10.135142 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:10.135192 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:10.139147 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:10.140302 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:10.140311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:10.140317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:10.140492 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:10.141114 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:10.141120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:10.141123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:10.141628 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:10.141860 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:10.142831 (+ 971us) server_negotiation.cc:299] Negotiation successful
0807 11:30:10.143148 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":224,"threads_started":1}
I20260807 11:30:10.145771 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:10.118976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.65:45371)
0807 11:30:10.119850 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:10.119916 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:10.122726 (+ 2810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:10.123626 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:10.123638 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:10.124435 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:10.125283 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:10.125310 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:10.126750 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:10.126757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:10.129034 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:10.129053 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:10.130228 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:10.130237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:10.130588 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:10.131468 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:10.131503 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:10.134805 (+ 3302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:10.139317 (+ 4512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:10.139329 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:10.139351 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:10.140068 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:10.140653 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:10.140660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:10.140665 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:10.140958 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:10.141822 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:10.141832 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:10.142344 (+ 512us) client_negotiation.cc:769] Sending connection context
0807 11:30:10.142681 (+ 337us) client_negotiation.cc:240] Negotiation successful
0807 11:30:10.143338 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":312,"threads_started":1}
I20260807 11:30:10.148145 32313 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42143
I20260807 11:30:10.148511 32313 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:10.149515 32313 heartbeater.cc:511] Master 127.24.103.126:42143 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260807 11:30:10.152315 32101 ts_manager.cc:194] Registered new tserver with Master: 7422a6ab063542509f75e3098e0dca05 (127.24.103.65:46611)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.66@KRBTEST.COM" created.
I20260807 11:30:10.154253 32101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:45371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
I20260807 11:30:10.190722 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42143
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:10.543537 32321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:10.543838 32321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:10.543936 32321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:10.551426 32321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:10.551578 32321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:10.551805 32321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:30:10.562549 32321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:10.564075 32321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:10.565383 32321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:10.578804 32326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:10.578840 32329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:10.578882 32327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:10.579696 32321 server_base.cc:1034] running on GCE node
I20260807 11:30:10.580266 32321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:10.581141 32321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:10.582348 32321 hybrid_clock.cc:648] HybridClock initialized: now 1786102210582305 us; error 36 us; skew 500 ppm
I20260807 11:30:10.582669 32321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:10 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102210, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:10.585915 32321 init.cc:377] Logged in from keytab as oryx/127.24.103.66@KRBTEST.COM (short username oryx)
I20260807 11:30:10.587527 32321 webserver.cc:460] Webserver started at http://127.24.103.66:42891/ using document root <none> and password file <none>
I20260807 11:30:10.588218 32321 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:10.588361 32321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:10.588654 32321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:10.591552 32321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "62c004b483df4454b90ba9594e4625e8"
format_stamp: "Formatted at 2026-08-07 11:30:10 on dist-test-slave-wl2h"
server_key: "fb4bc0aeca468517b5dcc0f28d47680c"
server_key_iv: "842ba04365221918581e1534b59cc568"
server_key_version: "encryptionkey@0"
I20260807 11:30:10.592432 32321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "62c004b483df4454b90ba9594e4625e8"
format_stamp: "Formatted at 2026-08-07 11:30:10 on dist-test-slave-wl2h"
server_key: "fb4bc0aeca468517b5dcc0f28d47680c"
server_key_iv: "842ba04365221918581e1534b59cc568"
server_key_version: "encryptionkey@0"
I20260807 11:30:10.597747 32321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:30:10.601358 32336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:10.602701 32321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 11:30:10.602871 32321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "62c004b483df4454b90ba9594e4625e8"
format_stamp: "Formatted at 2026-08-07 11:30:10 on dist-test-slave-wl2h"
server_key: "fb4bc0aeca468517b5dcc0f28d47680c"
server_key_iv: "842ba04365221918581e1534b59cc568"
server_key_version: "encryptionkey@0"
I20260807 11:30:10.603118 32321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:10.615585 32321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:10.619932 32321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:10.620226 32321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:10.621158 32321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:10.622488 32321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:10.622591 32321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:10.622735 32321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:10.622805 32321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:10.658954 32321 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:34573
I20260807 11:30:10.659000 32464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:34573 every 8 connection(s)
I20260807 11:30:10.660287 32321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:30:10.667781 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 32321
I20260807 11:30:10.667972 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:30:10.668453 24989 external_mini_cluster.cc:1428] Setting key d161ea84e06caf3d9ff6ead8a76d4226
Aug 07 11:30:10 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102210, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:10.681144 32467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:10.662922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:34133 (local address 127.24.103.126:42143)
0807 11:30:10.663443 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:10.663448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:10.664342 (+ 894us) server_negotiation.cc:407] Connection header received
0807 11:30:10.666364 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:10.666370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:10.666463 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:10.666589 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:10.668351 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:10.669206 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:10.670899 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:10.671447 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:10.671597 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:10.675288 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:10.675293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:10.675296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:10.675328 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:10.677882 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:10.678619 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:10.678623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:10.678625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:10.678761 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:10.679279 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:10.679283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:10.679285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:10.679651 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:10.679815 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:10.680499 (+ 684us) server_negotiation.cc:299] Negotiation successful
0807 11:30:10.680728 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":304,"threads_started":1}
I20260807 11:30:10.682142 32468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:10.663388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.66:34133)
0807 11:30:10.664159 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:10.664200 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:10.666073 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:10.666843 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:10.666851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:10.667461 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:10.668168 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:10.668189 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:10.669375 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:10.669381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:10.670725 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:10.670738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:10.671704 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:10.671714 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:10.671968 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:10.672734 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:10.672758 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:10.674992 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:10.678023 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:10.678033 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:10.678048 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:10.678480 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:10.678873 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:10.678878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:10.678881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:10.679154 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:10.679785 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:10.679793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:10.680144 (+ 351us) client_negotiation.cc:769] Sending connection context
0807 11:30:10.680405 (+ 261us) client_negotiation.cc:240] Negotiation successful
0807 11:30:10.680957 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":310,"threads_started":1}
I20260807 11:30:10.683625 32465 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42143
I20260807 11:30:10.683976 32465 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:10.684895 32465 heartbeater.cc:511] Master 127.24.103.126:42143 requested a full tablet report, sending...
I20260807 11:30:10.686673 32101 ts_manager.cc:194] Registered new tserver with Master: 62c004b483df4454b90ba9594e4625e8 (127.24.103.66:34573)
I20260807 11:30:10.687736 32101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:34133
WARNING: no policy specified for oryx/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
I20260807 11:30:10.731324 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42143
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:11.077427 32471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:11.077735 32471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:11.077896 32471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:11.085642 32471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:11.085780 32471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:11.085965 32471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:30:11.096956 32471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:11.098441 32471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:11.100035 32471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:11.113734 32477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:11.113741 32476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:11.114467 32479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:11.114704 32471 server_base.cc:1034] running on GCE node
I20260807 11:30:11.115485 32471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:11.116650 32471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:11.117875 32471 hybrid_clock.cc:648] HybridClock initialized: now 1786102211117834 us; error 61 us; skew 500 ppm
I20260807 11:30:11.118222 32471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:11 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102211, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:11.122133 32471 init.cc:377] Logged in from keytab as oryx/127.24.103.67@KRBTEST.COM (short username oryx)
I20260807 11:30:11.123756 32471 webserver.cc:460] Webserver started at http://127.24.103.67:44735/ using document root <none> and password file <none>
I20260807 11:30:11.124445 32471 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:11.124577 32471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:11.124887 32471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:11.127940 32471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "eae8abbb0c9242f49c9dd17ad13de1a6"
format_stamp: "Formatted at 2026-08-07 11:30:11 on dist-test-slave-wl2h"
server_key: "16f99554a8ae0f74bcb9df6c43f1071b"
server_key_iv: "db758143e5dd7d985254621fc4cd58ba"
server_key_version: "encryptionkey@0"
I20260807 11:30:11.128737 32471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "eae8abbb0c9242f49c9dd17ad13de1a6"
format_stamp: "Formatted at 2026-08-07 11:30:11 on dist-test-slave-wl2h"
server_key: "16f99554a8ae0f74bcb9df6c43f1071b"
server_key_iv: "db758143e5dd7d985254621fc4cd58ba"
server_key_version: "encryptionkey@0"
I20260807 11:30:11.134086 32471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:30:11.137709 32486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:11.139359 32471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 11:30:11.139549 32471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "eae8abbb0c9242f49c9dd17ad13de1a6"
format_stamp: "Formatted at 2026-08-07 11:30:11 on dist-test-slave-wl2h"
server_key: "16f99554a8ae0f74bcb9df6c43f1071b"
server_key_iv: "db758143e5dd7d985254621fc4cd58ba"
server_key_version: "encryptionkey@0"
I20260807 11:30:11.139782 32471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:11.158595 32313 heartbeater.cc:503] Master 127.24.103.126:42143 was elected leader, sending a full tablet report...
I20260807 11:30:11.168735 32471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:11.173601 32471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:11.173935 32471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:11.174971 32471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:11.176436 32471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:11.176548 32471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:11.176689 32471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:11.176761 32471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:11.211841 32471 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:37329
I20260807 11:30:11.211895 32611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:37329 every 8 connection(s)
I20260807 11:30:11.213250 32471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:30:11.220273 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 32471
I20260807 11:30:11.220476 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:30:11.220981 24989 external_mini_cluster.cc:1428] Setting key 3cd3bf7e8284255e9693f54669db2d31
Aug 07 11:30:11 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102211, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:11.233402 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.215959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:55689 (local address 127.24.103.126:42143)
0807 11:30:11.216426 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:11.216431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:11.217402 (+ 971us) server_negotiation.cc:407] Connection header received
0807 11:30:11.219311 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:11.219317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:11.219414 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:11.219531 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:11.221154 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.221940 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.223526 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.223942 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.224072 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.227487 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:11.227492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:11.227495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:11.227528 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:11.230093 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.230971 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.230976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.230978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.231151 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.231587 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.231592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.231595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.231883 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:11.232050 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:11.232755 (+ 705us) server_negotiation.cc:299] Negotiation successful
0807 11:30:11.232981 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":260,"threads_started":1}
I20260807 11:30:11.234341 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.216509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.67:55689)
0807 11:30:11.217242 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:11.217282 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:11.218983 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:11.219777 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:11.219785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:11.220291 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:11.220940 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.220957 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.222090 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.222095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.223387 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.223400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.224178 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.224185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.224405 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.225042 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:11.225064 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:11.227282 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:11.230283 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.230293 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.230310 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.230841 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.231261 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.231266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.231270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.231465 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.232021 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:11.232029 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:11.232382 (+ 353us) client_negotiation.cc:769] Sending connection context
0807 11:30:11.232652 (+ 270us) client_negotiation.cc:240] Negotiation successful
0807 11:30:11.233188 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":278,"threads_started":1}
I20260807 11:30:11.236207 32612 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42143
I20260807 11:30:11.236595 32612 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:11.237496 32612 heartbeater.cc:511] Master 127.24.103.126:42143 requested a full tablet report, sending...
I20260807 11:30:11.238950 32100 ts_manager.cc:194] Registered new tserver with Master: eae8abbb0c9242f49c9dd17ad13de1a6 (127.24.103.67:37329)
I20260807 11:30:11.240016 32100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:55689
I20260807 11:30:11.249640 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.103.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.125.keytab.
Entry for principal oryx/127.24.103.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.125.keytab.
I20260807 11:30:11.311344 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:35933
--webserver_interface=127.24.103.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.125.keytab
--principal=oryx/127.24.103.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42143,127.24.103.125:35933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:11.662166 32619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:11.662533 32619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:11.662722 32619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:11.670287 32619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:11.670439 32619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:11.670522 32619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:11.670605 32619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:11.670670 32619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:30:11.681041 32619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42143,127.24.103.125:35933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.125.keytab
--principal=oryx/127.24.103.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:35933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.103.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
W20260807 11:30:11.681496 32619 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.103.126:42143,127.24.103.125:35933'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 11:30:11.682560 32619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:11.683838 32619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 11:30:11.691109 32465 heartbeater.cc:503] Master 127.24.103.126:42143 was elected leader, sending a full tablet report...
W20260807 11:30:11.696324 32625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:11.696436 32628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:11.696481 32626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:11.697863 32619 server_base.cc:1034] running on GCE node
I20260807 11:30:11.698655 32619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:11.700146 32619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:11.701403 32619 hybrid_clock.cc:648] HybridClock initialized: now 1786102211701339 us; error 34 us; skew 500 ppm
I20260807 11:30:11.701761 32619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:11 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102211, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:11.705416 32619 init.cc:377] Logged in from keytab as oryx/127.24.103.125@KRBTEST.COM (short username oryx)
I20260807 11:30:11.707305 32619 webserver.cc:460] Webserver started at http://127.24.103.125:36999/ using document root <none> and password file <none>
I20260807 11:30:11.708040 32619 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:11.708173 32619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:11.708503 32619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:11.711890 32619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data/instance:
uuid: "53f447304fdb45bf9a9f7ddf07bf91d3"
format_stamp: "Formatted at 2026-08-07 11:30:11 on dist-test-slave-wl2h"
server_key: "f6224fe260cc850543e94e3ca2a0689b"
server_key_iv: "fb2ce9dac60bb0c9f45f99a9df8aaf24"
server_key_version: "encryptionkey@0"
I20260807 11:30:11.712739 32619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/wal/instance:
uuid: "53f447304fdb45bf9a9f7ddf07bf91d3"
format_stamp: "Formatted at 2026-08-07 11:30:11 on dist-test-slave-wl2h"
server_key: "f6224fe260cc850543e94e3ca2a0689b"
server_key_iv: "fb2ce9dac60bb0c9f45f99a9df8aaf24"
server_key_version: "encryptionkey@0"
I20260807 11:30:11.718636 32619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:30:11.722446 32637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:11.724081 32619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 11:30:11.724308 32619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/wal
uuid: "53f447304fdb45bf9a9f7ddf07bf91d3"
format_stamp: "Formatted at 2026-08-07 11:30:11 on dist-test-slave-wl2h"
server_key: "f6224fe260cc850543e94e3ca2a0689b"
server_key_iv: "fb2ce9dac60bb0c9f45f99a9df8aaf24"
server_key_version: "encryptionkey@0"
I20260807 11:30:11.724565 32619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:11.741860 32619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:11.746668 32619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:11.746960 32619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:11.768316 32619 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.125:35933
I20260807 11:30:11.768335 32693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.125:35933 every 8 connection(s)
I20260807 11:30:11.769884 32619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
I20260807 11:30:11.774034 32694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:11.778273 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 32619
I20260807 11:30:11.782122 32694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:30:11 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
Aug 07 11:30:11 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102211, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.125@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:11.803870 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.786663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.24.103.126:42143)
0807 11:30:11.787212 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:11.787220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:11.787769 (+ 549us) server_negotiation.cc:407] Connection header received
0807 11:30:11.787985 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:11.787989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:11.788068 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:11.788185 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:11.789708 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.790692 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.792688 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.793127 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.793232 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.797227 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:11.797234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:11.797239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:11.797286 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:11.800851 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.801566 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.801571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.801575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.801706 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.802104 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.802109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.802112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.802448 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:11.802614 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:11.803277 (+ 663us) server_negotiation.cc:299] Negotiation successful
0807 11:30:11.803514 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":274,"threads_started":1}
I20260807 11:30:11.804930 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.786786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:35430)
0807 11:30:11.787627 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:11.787662 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:11.787851 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:11.788296 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:11.788303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:11.789182 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:11.789516 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.789529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.790843 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.790847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.792537 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.792554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.793374 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.793384 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.793629 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.794345 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:11.794379 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:11.796929 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:11.800979 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.800990 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.801003 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.801437 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.801816 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.801821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.801824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.801985 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.802578 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:11.802586 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:11.802848 (+ 262us) client_negotiation.cc:769] Sending connection context
0807 11:30:11.803184 (+ 336us) client_negotiation.cc:240] Negotiation successful
0807 11:30:11.803851 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":186,"thread_start_us":485,"threads_started":1}
I20260807 11:30:11.804908 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.780854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.24.103.125:35933)
0807 11:30:11.781745 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:11.781758 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:11.781803 (+ 45us) server_negotiation.cc:407] Connection header received
0807 11:30:11.783657 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:11.783682 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:11.784149 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:11.784553 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:11.786571 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.788669 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.790297 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.791256 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.791401 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.796758 (+ 5357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:11.796771 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:11.796787 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:11.796828 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:11.800608 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.801346 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.801360 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.801362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.801529 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.801950 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.801956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.801959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.802495 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:11.802833 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:11.803205 (+ 372us) server_negotiation.cc:299] Negotiation successful
0807 11:30:11.803707 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":668,"thread_start_us":450,"threads_started":1}
I20260807 11:30:11.807106 32694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:30:11 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102211, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.125@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
I20260807 11:30:11.820537 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.807914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:35933 (local address 127.0.0.1:54648)
0807 11:30:11.808076 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:11.808099 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:11.808194 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:11.808520 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:11.808524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:11.808864 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:11.808993 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.809001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.810115 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.810119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.811185 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.811198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.811947 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.811951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.812181 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.812795 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:11.812813 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:11.815097 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:11.818106 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.818110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.818113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.818465 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.818862 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.818866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.818868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.819008 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.819773 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:11.819777 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:11.819905 (+ 128us) client_negotiation.cc:769] Sending connection context
0807 11:30:11.820018 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 11:30:11.820251 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 11:30:11.820633 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.808092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.24.103.125:35933)
0807 11:30:11.808243 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:11.808246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:11.808260 (+ 14us) server_negotiation.cc:407] Connection header received
0807 11:30:11.808297 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:11.808300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:11.808348 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:11.808423 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:11.809190 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.809891 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.811332 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.811796 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.811916 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.815308 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:11.815313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:11.815316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:11.815346 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:11.817969 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.818594 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.818599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.818601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.818731 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.819299 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.819303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.819305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.819613 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:11.819757 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:11.820227 (+ 470us) server_negotiation.cc:299] Negotiation successful
0807 11:30:11.820396 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 11:30:11.825927 32694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: Bootstrap starting.
I20260807 11:30:11.829360 32694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:11.830488 32694 log.cc:824] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:11.833614 32694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: No bootstrap required, opened a new log
I20260807 11:30:11.841022 32694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } }
I20260807 11:30:11.841429 32694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:11.841534 32694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f447304fdb45bf9a9f7ddf07bf91d3, State: Initialized, Role: FOLLOWER
I20260807 11:30:11.842177 32694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } }
I20260807 11:30:11.843212 32701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } } }
I20260807 11:30:11.843592 32701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:30:11.844408 32694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 11:30:11.855062 32714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:30:11.855273 32714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:30:11.857815 32619 master_runner.cc:428] Detected that this master 53f447304fdb45bf9a9f7ddf07bf91d3 is joining an existing cluster
I20260807 11:30:11.857947 32619 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.103.125:35933
I20260807 11:30:11.887596 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.867301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.24.103.125:35933)
0807 11:30:11.867540 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:11.867549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:11.869441 (+ 1892us) server_negotiation.cc:407] Connection header received
0807 11:30:11.869662 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:11.869667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:11.869747 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:11.869865 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:11.871683 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.873171 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.875456 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.876374 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.876492 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.879452 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:11.879461 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:11.879469 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:11.879578 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:11.883802 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.884696 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.884704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.884711 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.884901 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.885363 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.885370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.885376 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.885785 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:11.886058 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:11.886683 (+ 625us) server_negotiation.cc:299] Negotiation successful
0807 11:30:11.886951 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48,"spinlock_wait_cycles":28928}
I20260807 11:30:11.889171 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.866825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:35933 (local address 127.0.0.1:54664)
0807 11:30:11.869294 (+ 2469us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:11.869319 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:11.869510 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:11.870033 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:11.870037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:11.870821 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:11.871495 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.871510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.873376 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.873381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.875294 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.875312 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.876883 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.876893 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.877129 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.877930 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:11.877961 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:11.879248 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:11.883965 (+ 4717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.883973 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.883979 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.884535 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.885059 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.885066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.885072 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.885246 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.886128 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:11.886135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:11.886377 (+ 242us) client_negotiation.cc:769] Sending connection context
0807 11:30:11.888247 (+ 1870us) client_negotiation.cc:240] Negotiation successful
0807 11:30:11.888612 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2282,"thread_start_us":2099,"threads_started":1}
I20260807 11:30:11.890204 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.866801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.24.103.126:42143)
0807 11:30:11.867061 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:11.867071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:11.871774 (+ 4703us) server_negotiation.cc:407] Connection header received
0807 11:30:11.872077 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:11.872085 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:11.872207 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:11.872285 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:11.873917 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.875189 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.877947 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.878689 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.878828 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.882443 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:11.882451 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:11.882457 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:11.882509 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:11.886614 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.887577 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.887584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.887588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.887776 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.888308 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.888314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.888318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.888706 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:11.888853 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:11.889545 (+ 692us) server_negotiation.cc:299] Negotiation successful
0807 11:30:11.889826 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:30:11.891388 32721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.867520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:35440)
0807 11:30:11.871642 (+ 4122us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:11.871664 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:11.871901 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:11.872519 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:11.872526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:11.873348 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:11.873722 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.873732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.875414 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.875421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.877333 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.877351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.879219 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.879227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.879444 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.880170 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:11.880195 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:11.882178 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:11.886825 (+ 4647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.886830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.886834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.887429 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.887946 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.887953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.887963 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.888154 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.888960 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:11.888966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:11.889211 (+ 245us) client_negotiation.cc:769] Sending connection context
0807 11:30:11.890471 (+ 1260us) client_negotiation.cc:240] Negotiation successful
0807 11:30:11.890877 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3975,"thread_start_us":3739,"threads_started":1}
WARNING: no policy specified for oryx/127.24.103.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.124@KRBTEST.COM" created.
Aug 07 11:30:11 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102209, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.126@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
I20260807 11:30:11.929144 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.899538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.24.103.125:35933)
0807 11:30:11.899752 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:11.899760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:11.908341 (+ 8581us) server_negotiation.cc:407] Connection header received
0807 11:30:11.908720 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:11.908728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:11.908864 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:11.908977 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:11.911104 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.912333 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.914580 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.915486 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.915628 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.920797 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:11.920807 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:11.920814 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:11.920863 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:11.925380 (+ 4517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.926555 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.926564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.926570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.926731 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:11.927316 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:11.927321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:11.927323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:11.927676 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:11.927798 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:11.928525 (+ 727us) server_negotiation.cc:299] Negotiation successful
0807 11:30:11.928853 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:30:11.929368 32723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:11.899514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:35933 (local address 127.0.0.1:54670)
0807 11:30:11.908193 (+ 8679us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:11.908230 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:11.908446 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:11.909457 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:11.909472 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:11.910482 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:11.910844 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.910860 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.912524 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.912531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.914407 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:11.914425 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:11.915911 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:11.915922 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:11.916181 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:11.917070 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:11.917098 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:11.920539 (+ 3441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:11.925586 (+ 5047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.925607 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.925617 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.926362 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.926875 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:11.926881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:11.926886 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:11.927143 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:11.927896 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:11.927907 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:11.928186 (+ 279us) client_negotiation.cc:769] Sending connection context
0807 11:30:11.928422 (+ 236us) client_negotiation.cc:240] Negotiation successful
0807 11:30:11.928884 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8382,"thread_start_us":8087,"threads_started":1}
I20260807 11:30:11.931883 32100 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.103.125:35933: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3"
member_type: NON_VOTER
last_known_addr {
host: "127.24.103.125"
port: 35933
}
attrs {
promote: true
}
}
dest_uuid: "7182993a35ce4f5aa488a3caa7acea01"
cas_config_opid_index: -1
I20260807 11:30:11.933244 32100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.124.keytab.
Entry for principal oryx/127.24.103.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.124.keytab.
I20260807 11:30:11.935881 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:36941
--webserver_interface=127.24.103.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.124.keytab
--principal=oryx/127.24.103.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42143,127.24.103.125:35933,127.24.103.124:36941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:30:11.948360 32726 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 53f447304fdb45bf9a9f7ddf07bf91d3 (127.24.103.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } } }
I20260807 11:30:11.949397 32667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:11.953788 32667 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 FOLLOWER]: Refusing update from remote peer 7182993a35ce4f5aa488a3caa7acea01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260807 11:30:11.953822 32726 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.103.125:35933
I20260807 11:30:11.956130 32726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:11.956554 32619 master.cc:537] Master@127.24.103.125:35933 shutting down...
I20260807 11:30:11.956940 32727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } } }
I20260807 11:30:11.957356 32727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: This master's current role is: LEADER
W20260807 11:30:11.964231 32084 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260807 11:30:11.984607 32619 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:11.985196 32619 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:11.985446 32619 tablet_replica.cc:330] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: stopping tablet replica
W20260807 11:30:11.985965 32084 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 -> Peer 53f447304fdb45bf9a9f7ddf07bf91d3 (127.24.103.125:35933): Couldn't send request to peer 53f447304fdb45bf9a9f7ddf07bf91d3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 11:30:11.994364 32619 master.cc:552] Master@127.24.103.125:35933 shutdown complete.
I20260807 11:30:11.994513 32619 master_runner.cc:305] Clearing existing system tablet
I20260807 11:30:11.994994 32619 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:11.995138 32619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:11.998330 32619 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 11:30:12.000038 32734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:12.000442 32619 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:12.000718 32619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 11:30:12.000806 32619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/wal
uuid: "53f447304fdb45bf9a9f7ddf07bf91d3"
format_stamp: "Formatted at 2026-08-07 11:30:11 on dist-test-slave-wl2h"
server_key: "f6224fe260cc850543e94e3ca2a0689b"
server_key_iv: "fb2ce9dac60bb0c9f45f99a9df8aaf24"
server_key_version: "encryptionkey@0"
I20260807 11:30:12.005936 32619 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:12.013237 32619 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 11:30:12.013441 32619 log.cc:1196] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260807 11:30:12.013904 32619 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: Deleting consensus metadata
I20260807 11:30:12.014204 32619 master_runner.cc:315] Copying system tablet from 127.24.103.126:42143
I20260807 11:30:12.017258 32619 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: tablet copy: Beginning tablet copy session from remote peer at address 127.24.103.126:42143
I20260807 11:30:12.035033 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.018433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.24.103.126:42143)
0807 11:30:12.018627 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.018634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.019138 (+ 504us) server_negotiation.cc:407] Connection header received
0807 11:30:12.019350 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.019356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.019443 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.019563 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:12.020690 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.021922 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.023815 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.024504 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.024642 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.027222 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:12.027230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:12.027238 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:12.027288 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:12.031647 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.032547 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.032553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.032557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.032744 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.033225 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.033231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.033235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.033628 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:12.033788 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.034387 (+ 599us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.034628 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:30:12.035156 32743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.018398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:35450)
0807 11:30:12.018974 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:12.018999 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:12.019197 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:12.019607 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:12.019612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:12.020216 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:12.020506 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.020520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.022105 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.022112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.023646 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.023662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.024675 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.024681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.024879 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.025801 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:12.025827 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:12.026932 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:12.031814 (+ 4882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.031820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.031824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.032391 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.032873 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.032878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.032882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.033078 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.033821 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:12.033827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:12.034060 (+ 233us) client_negotiation.cc:769] Sending connection context
0807 11:30:12.034273 (+ 213us) client_negotiation.cc:240] Negotiation successful
0807 11:30:12.034637 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":330,"threads_started":1}
I20260807 11:30:12.036475 32126 tablet_copy_service.cc:140] P 7182993a35ce4f5aa488a3caa7acea01: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 53f447304fdb45bf9a9f7ddf07bf91d3 ({username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:35450)
I20260807 11:30:12.036964 32126 tablet_copy_service.cc:161] P 7182993a35ce4f5aa488a3caa7acea01: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 53f447304fdb45bf9a9f7ddf07bf91d3 at {username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:35450: session id = 53f447304fdb45bf9a9f7ddf07bf91d3-00000000000000000000000000000000
I20260807 11:30:12.041855 32126 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:30:12.044804 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 11:30:12.052167 32619 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: tablet copy: Starting download of 0 data blocks...
I20260807 11:30:12.052596 32619 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: tablet copy: Starting download of 1 WAL segments...
I20260807 11:30:12.056743 32619 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:30:12.061677 32126 tablet_copy_service.cc:342] P 7182993a35ce4f5aa488a3caa7acea01: Request end of tablet copy session 53f447304fdb45bf9a9f7ddf07bf91d3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:35450
I20260807 11:30:12.061956 32126 tablet_copy_service.cc:433] P 7182993a35ce4f5aa488a3caa7acea01: ending tablet copy session 53f447304fdb45bf9a9f7ddf07bf91d3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 53f447304fdb45bf9a9f7ddf07bf91d3
I20260807 11:30:12.074982 32619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:12.077852 32744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:12.083789 32745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:12.091609 32747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:12.092507 32619 server_base.cc:1034] running on GCE node
I20260807 11:30:12.093081 32619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:12.093844 32619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:12.095079 32619 hybrid_clock.cc:648] HybridClock initialized: now 1786102212095016 us; error 38 us; skew 500 ppm
I20260807 11:30:12.095360 32619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:12 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102212, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:12.097666 32619 init.cc:377] Logged in from keytab as oryx/127.24.103.125@KRBTEST.COM (short username oryx)
I20260807 11:30:12.099962 32619 webserver.cc:460] Webserver started at http://127.24.103.125:41981/ using document root <none> and password file <none>
I20260807 11:30:12.100478 32619 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:12.100577 32619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:12.103928 32619 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260807 11:30:12.106261 32754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:12.106840 32619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 11:30:12.107076 32619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/wal
uuid: "53f447304fdb45bf9a9f7ddf07bf91d3"
format_stamp: "Formatted at 2026-08-07 11:30:11 on dist-test-slave-wl2h"
server_key: "f6224fe260cc850543e94e3ca2a0689b"
server_key_iv: "fb2ce9dac60bb0c9f45f99a9df8aaf24"
server_key_version: "encryptionkey@0"
I20260807 11:30:12.107358 32619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:12.132887 32619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:12.137284 32619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:12.171334 32619 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.125:35933
I20260807 11:30:12.171378 341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.125:35933 every 8 connection(s)
I20260807 11:30:12.172500 32619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
I20260807 11:30:12.178093 342 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:30:12.182169 342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: Bootstrap starting.
I20260807 11:30:12.201090 342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:30:12.201769 342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: Bootstrap complete.
I20260807 11:30:12.202988 342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } }
I20260807 11:30:12.203348 342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 53f447304fdb45bf9a9f7ddf07bf91d3, State: Initialized, Role: LEARNER
I20260807 11:30:12.203655 342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } }
I20260807 11:30:12.204924 345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } } }
I20260807 11:30:12.205224 342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:12.205451 345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: This master's current role is: LEARNER
I20260807 11:30:12.228555 357 catalog_manager.cc:1259] Loaded cluster ID: 13e0d15ed169414d8d7b9cb933658be1
I20260807 11:30:12.228719 357 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: loading cluster ID for follower catalog manager: success
I20260807 11:30:12.233497 357 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: acquiring CA information for follower catalog manager: success
I20260807 11:30:12.236369 357 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:30:12.242848 32612 heartbeater.cc:503] Master 127.24.103.126:42143 was elected leader, sending a full tablet report...
I20260807 11:30:12.392263 32723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.382325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:35933 (local address 127.0.0.1:54680)
0807 11:30:12.382593 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:12.382614 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:12.382746 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:12.383816 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:12.383823 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:12.383881 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:12.384132 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.384147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.386487 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.386495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.389061 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.389082 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.391339 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.391346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.391576 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.391643 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 11:30:12.391761 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 11:30:12.391915 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260807 11:30:12.392762 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.382564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.24.103.125:35933)
0807 11:30:12.383379 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.383385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.383413 (+ 28us) server_negotiation.cc:407] Connection header received
0807 11:30:12.383498 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.383503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.383590 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.383735 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:12.384356 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.386286 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.389397 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.391140 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.391372 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.391571 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.391814 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.391972 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":449,"threads_started":1}
I20260807 11:30:12.394060 316 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260807 11:30:12.394894 345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7182993a35ce4f5aa488a3caa7acea01. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } } }
I20260807 11:30:12.395249 345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: This master's current role is: LEARNER
W20260807 11:30:12.409200 32728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:12.409588 32728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:12.409788 32728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:12.418534 32728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:12.418659 32728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:12.418763 32728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:12.418848 32728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:12.418910 32728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:30:12.430655 32728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42143,127.24.103.125:35933,127.24.103.124:36941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.124.keytab
--principal=oryx/127.24.103.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:36941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.103.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:12.432264 32728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:12.433755 32728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:12.446802 366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:12.447188 365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:12.447607 368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:12.447906 32728 server_base.cc:1034] running on GCE node
I20260807 11:30:12.448689 32728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:12.450169 32728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:12.451416 32728 hybrid_clock.cc:648] HybridClock initialized: now 1786102212451367 us; error 32 us; skew 500 ppm
I20260807 11:30:12.451817 32728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:12 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102212, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:12.455741 32728 init.cc:377] Logged in from keytab as oryx/127.24.103.124@KRBTEST.COM (short username oryx)
I20260807 11:30:12.457592 32728 webserver.cc:460] Webserver started at http://127.24.103.124:46307/ using document root <none> and password file <none>
I20260807 11:30:12.458377 32728 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:12.458549 32728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:12.458904 32728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:12.462433 32728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data/instance:
uuid: "53489d42e48a489eb63b33109c42b145"
format_stamp: "Formatted at 2026-08-07 11:30:12 on dist-test-slave-wl2h"
server_key: "3c403a1dd363d0b86811972f30d30fa5"
server_key_iv: "41dcffe97c4c83e93ee0781f7561ef38"
server_key_version: "encryptionkey@0"
I20260807 11:30:12.463434 32728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/wal/instance:
uuid: "53489d42e48a489eb63b33109c42b145"
format_stamp: "Formatted at 2026-08-07 11:30:12 on dist-test-slave-wl2h"
server_key: "3c403a1dd363d0b86811972f30d30fa5"
server_key_iv: "41dcffe97c4c83e93ee0781f7561ef38"
server_key_version: "encryptionkey@0"
I20260807 11:30:12.469053 32728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:30:12.473202 375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:12.474763 32728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:30:12.474982 32728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/wal
uuid: "53489d42e48a489eb63b33109c42b145"
format_stamp: "Formatted at 2026-08-07 11:30:12 on dist-test-slave-wl2h"
server_key: "3c403a1dd363d0b86811972f30d30fa5"
server_key_iv: "41dcffe97c4c83e93ee0781f7561ef38"
server_key_version: "encryptionkey@0"
I20260807 11:30:12.475286 32728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:12.501705 32728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:12.506804 32728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:12.507229 32728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:12.528141 32728 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.124:36941
I20260807 11:30:12.528188 429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.124:36941 every 8 connection(s)
I20260807 11:30:12.529619 32728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
I20260807 11:30:12.532692 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 32728
I20260807 11:30:12.534708 430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:12.545162 430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:30:12 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
Aug 07 11:30:12 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102212, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.124@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:12.560330 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.535275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.24.103.124:36941)
0807 11:30:12.536095 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.536106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.536149 (+ 43us) server_negotiation.cc:407] Connection header received
0807 11:30:12.537773 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.537795 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.538187 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.538542 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:12.540753 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.542310 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.544410 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.545226 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.545370 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.552108 (+ 6738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:12.552115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:12.552125 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:12.552167 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:12.555654 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.556502 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.556510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.556518 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.556692 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.557276 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.557280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.557282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.557813 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:12.558065 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.558597 (+ 532us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.559095 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":378,"threads_started":1}
I20260807 11:30:12.565913 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.549376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:35462)
0807 11:30:12.550530 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:12.550581 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:12.550797 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:12.551344 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:12.551353 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:12.552240 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:12.552566 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.552576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.554072 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.554080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.555542 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.555555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.556408 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.556418 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.556586 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.557299 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:12.557325 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:12.560062 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:12.563568 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.563576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.563581 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.564001 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.564409 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.564412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.564414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.564534 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.565049 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:12.565054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:12.565230 (+ 176us) client_negotiation.cc:769] Sending connection context
0807 11:30:12.565390 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 11:30:12.565626 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":628,"threads_started":1}
I20260807 11:30:12.566010 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.549443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.24.103.126:42143)
0807 11:30:12.550091 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.550098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.550748 (+ 650us) server_negotiation.cc:407] Connection header received
0807 11:30:12.550968 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.550973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.551096 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.551203 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:12.552745 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.553918 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.555710 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.556190 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.556300 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.560269 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:12.560279 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:12.560286 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:12.560333 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:12.563442 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.564164 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.564168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.564171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.564309 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.564641 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.564644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.564646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.564934 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:12.565049 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.565494 (+ 445us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.565679 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":283,"threads_started":1}
I20260807 11:30:12.567237 430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:30:12 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102212, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.124@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
I20260807 11:30:12.579589 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.567958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:35933 (local address 127.0.0.1:54694)
0807 11:30:12.568167 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:12.568187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:12.568284 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:12.568569 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:12.568573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:12.568928 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:12.569060 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.569068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.570076 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.570084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.571071 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.571084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.571780 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.571787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.571958 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.572504 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:12.572520 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:12.574524 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:12.577394 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.577400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.577404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.577780 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.578143 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.578147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.578151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.578282 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.578781 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:12.578786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:12.578933 (+ 147us) client_negotiation.cc:769] Sending connection context
0807 11:30:12.579119 (+ 186us) client_negotiation.cc:240] Negotiation successful
0807 11:30:12.579358 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 11:30:12.580219 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.568074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54694 (local address 127.24.103.125:35933)
0807 11:30:12.568278 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.568284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.568303 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:30:12.568379 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.568384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.568451 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.568550 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:12.569202 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.569953 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.571203 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.571612 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.571750 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.574714 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:12.574720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:12.574725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:12.574760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:12.577263 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.577903 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.577913 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.577915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.578035 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.578387 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.578392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.578396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.578662 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:12.578821 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.579399 (+ 578us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.579899 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 11:30:12.581398 430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:30:12 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102212, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.124@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
I20260807 11:30:12.595096 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.582139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36941 (local address 127.0.0.1:55094)
0807 11:30:12.582357 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:12.582379 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:12.582516 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:12.582820 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:12.582825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:12.583307 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:12.583466 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.583474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.584414 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.584422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.585256 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.585267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.585992 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.585997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.586182 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.586811 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:12.586828 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:12.588925 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:12.592544 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.592549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.592553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.593005 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.593347 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.593353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.593356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.593503 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.594023 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:12.594027 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:12.594183 (+ 156us) client_negotiation.cc:769] Sending connection context
0807 11:30:12.594317 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 11:30:12.594614 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260807 11:30:12.595146 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.582161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55094 (local address 127.24.103.124:36941)
0807 11:30:12.582357 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.582364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.582455 (+ 91us) server_negotiation.cc:407] Connection header received
0807 11:30:12.582637 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.582644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.582704 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.582805 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:12.583590 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.584294 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.585387 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.585835 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.585977 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.589100 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:12.589104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:12.589107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:12.589135 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:12.592401 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.593114 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.593121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.593124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.593244 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.593623 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.593629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.593632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.593903 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:12.594060 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.594394 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.594763 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":73}
I20260807 11:30:12.600260 430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: Bootstrap starting.
I20260807 11:30:12.603567 430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:12.604590 430 log.cc:824] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:12.608211 430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: No bootstrap required, opened a new log
I20260807 11:30:12.616254 430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } }
I20260807 11:30:12.616608 430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:12.616755 430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53489d42e48a489eb63b33109c42b145, State: Initialized, Role: FOLLOWER
I20260807 11:30:12.617404 430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } }
I20260807 11:30:12.618455 437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } } }
I20260807 11:30:12.618861 437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:30:12.619546 430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:12.627892 437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:30:12.628260 437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } }
W20260807 11:30:12.629565 449 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:30:12.629709 449 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:30:12.629915 437 leader_election.cc:288] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7182993a35ce4f5aa488a3caa7acea01 (127.24.103.126:42143), 53f447304fdb45bf9a9f7ddf07bf91d3 (127.24.103.125:35933)
I20260807 11:30:12.630497 316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53489d42e48a489eb63b33109c42b145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" is_pre_election: true
I20260807 11:30:12.630482 32113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53489d42e48a489eb63b33109c42b145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7182993a35ce4f5aa488a3caa7acea01" is_pre_election: true
I20260807 11:30:12.631582 376 leader_election.cc:302] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53489d42e48a489eb63b33109c42b145; no voters: 53f447304fdb45bf9a9f7ddf07bf91d3, 7182993a35ce4f5aa488a3caa7acea01
I20260807 11:30:12.632004 437 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 11:30:12.634924 32728 master_runner.cc:428] Detected that this master 53489d42e48a489eb63b33109c42b145 is joining an existing cluster
I20260807 11:30:12.635133 32728 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.103.124:36941
I20260807 11:30:12.662959 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.641275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.24.103.124:36941)
0807 11:30:12.641505 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.641513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.644996 (+ 3483us) server_negotiation.cc:407] Connection header received
0807 11:30:12.645184 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.645192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.645266 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.645369 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:12.647428 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.648596 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.651043 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.651789 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.651877 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.655286 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:12.655291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:12.655296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:12.655334 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:12.659640 (+ 4306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.660503 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.660511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.660516 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.660658 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.661150 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.661157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.661163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.661572 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:12.661734 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.662283 (+ 549us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.662523 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:30:12.664223 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.641907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36941 (local address 127.0.0.1:55098)
0807 11:30:12.644893 (+ 2986us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:12.644912 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:12.645057 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:12.645552 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:12.645565 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:12.646633 (+ 1068us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:12.647236 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.647254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.648818 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.648822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.650861 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.650876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.652225 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.652233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.652461 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.653641 (+ 1180us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:12.653668 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:12.655065 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:12.659816 (+ 4751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.659820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.659825 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.660355 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.660809 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.660816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.660822 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.660998 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.661805 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:12.661810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:12.661989 (+ 179us) client_negotiation.cc:769] Sending connection context
0807 11:30:12.663279 (+ 1290us) client_negotiation.cc:240] Negotiation successful
0807 11:30:12.663671 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2837,"mutex_wait_us":203,"thread_start_us":2394,"threads_started":1}
I20260807 11:30:12.666678 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.639420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.24.103.126:42143)
0807 11:30:12.639634 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.639642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.644879 (+ 5237us) server_negotiation.cc:407] Connection header received
0807 11:30:12.645254 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.645260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.645348 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.645435 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:12.647460 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.648629 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.651049 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.651769 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.651877 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.658627 (+ 6750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:12.658635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:12.658641 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:12.658688 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:12.662466 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.663832 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.663837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.663842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.664005 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.664865 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.664870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.664873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.665256 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:12.665397 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.666039 (+ 642us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.666315 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 11:30:12.666886 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.640273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.24.103.125:35933)
0807 11:30:12.640516 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.640522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.644263 (+ 3741us) server_negotiation.cc:407] Connection header received
0807 11:30:12.645611 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.645617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.645709 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.645791 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:12.647671 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.648775 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.651469 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.652133 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.652246 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.655392 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:12.655398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:12.655402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:12.655446 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:12.659485 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.664284 (+ 4799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.664292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.664295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.664475 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.665007 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.665013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.665017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.665422 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:12.665562 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.666272 (+ 710us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.666543 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260807 11:30:12.668211 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 32172
I20260807 11:30:12.668391 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.639715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:35933 (local address 127.0.0.1:54704)
0807 11:30:12.644102 (+ 4387us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:12.644129 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:12.645441 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:12.646030 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:12.646036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:12.646927 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:12.647483 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.647495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.648982 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.648987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.651280 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.651294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.652574 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.652583 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.652787 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.653510 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:12.653533 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:12.655174 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:12.663587 (+ 8413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.663592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.663595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.664097 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.664644 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.664649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.664654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.664868 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.665646 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:12.665650 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:12.665920 (+ 270us) client_negotiation.cc:769] Sending connection context
0807 11:30:12.667504 (+ 1584us) client_negotiation.cc:240] Negotiation successful
0807 11:30:12.667844 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4186,"mutex_wait_us":29,"spinlock_wait_cycles":7424,"thread_start_us":3979,"threads_started":1}
I20260807 11:30:12.670969 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.639851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:35464)
0807 11:30:12.644729 (+ 4878us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:12.644753 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:12.644924 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:12.645599 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:12.645603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:12.646787 (+ 1184us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:12.647285 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.647298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.648817 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.648822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.650862 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.650876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.652259 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.652265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.652467 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.653329 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:12.653355 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:12.658316 (+ 4961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:12.662650 (+ 4334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.662655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.662659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.663667 (+ 1008us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.664163 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.664170 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.664173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.664714 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.665478 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:12.665482 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:12.665713 (+ 231us) client_negotiation.cc:769] Sending connection context
0807 11:30:12.668411 (+ 2698us) client_negotiation.cc:240] Negotiation successful
0807 11:30:12.670656 (+ 2245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4659,"mutex_wait_us":221,"spinlock_wait_cycles":29824,"thread_start_us":5307,"threads_started":1}
Aug 07 11:30:12 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.65@KRBTEST.COM
Aug 07 11:30:12 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102209, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.126@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
I20260807 11:30:12.692103 460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.669802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.24.103.65:46611)
0807 11:30:12.670484 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.670495 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.670529 (+ 34us) server_negotiation.cc:407] Connection header received
0807 11:30:12.670603 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.670612 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.670921 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.671646 (+ 725us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:12.673066 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.674723 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.676511 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.677427 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.677578 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.683653 (+ 6075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:12.683670 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:12.683692 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:12.683743 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:12.688136 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.689182 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.689194 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.689201 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.689380 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.689835 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.689840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.689843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.690308 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:12.690651 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.691296 (+ 645us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.691615 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":493,"threads_started":1}
I20260807 11:30:12.694007 32723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.676817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36941 (local address 127.0.0.1:55108)
0807 11:30:12.677226 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:12.677245 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:12.677356 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:12.678111 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:12.678120 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:12.678599 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:12.678811 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.678822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.680188 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.680194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.681242 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.681254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.682223 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.682230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.682419 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.683269 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:12.683294 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:12.686331 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:12.690724 (+ 4393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.690728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.690730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.691211 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.691666 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.691671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.691674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.691831 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.692954 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:12.692959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:12.693147 (+ 188us) client_negotiation.cc:769] Sending connection context
0807 11:30:12.693301 (+ 154us) client_negotiation.cc:240] Negotiation successful
0807 11:30:12.693612 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260807 11:30:12.694007 32307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:12.694948 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.677580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.24.103.124:36941)
0807 11:30:12.677790 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.677796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.677811 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:30:12.677893 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.677896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.677958 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.678086 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:12.678962 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.680063 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.681377 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.682019 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.682171 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.686575 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:12.686581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:12.686585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:12.686625 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:12.690559 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.691360 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.691365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.691368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.691514 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.692536 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.692547 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.692549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.692828 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:12.692991 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.694319 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.694593 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260807 11:30:12.697113 32100 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.103.124:36941: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "53489d42e48a489eb63b33109c42b145"
member_type: NON_VOTER
last_known_addr {
host: "127.24.103.124"
port: 36941
}
attrs {
promote: true
}
}
dest_uuid: "7182993a35ce4f5aa488a3caa7acea01"
cas_config_opid_index: 5
I20260807 11:30:12.698454 32100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } }
I20260807 11:30:12.700454 32727 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 53489d42e48a489eb63b33109c42b145 (127.24.103.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:12.704687 32727 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.103.124:36941
I20260807 11:30:12.705060 462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:12.705560 462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:12.705715 32728 master.cc:537] Master@127.24.103.124:36941 shutting down...
I20260807 11:30:12.706432 316 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 LEARNER]: Refusing update from remote peer 7182993a35ce4f5aa488a3caa7acea01: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 11:30:12.706717 403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:12.707403 32727 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 11:30:12.710979 316 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 53489d42e48a489eb63b33109c42b145 (127.24.103.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:12.711905 403 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 FOLLOWER]: Refusing update from remote peer 7182993a35ce4f5aa488a3caa7acea01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 11:30:12.712772 462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:12.715622 345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: NON_VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: true } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:12.715924 345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: This master's current role is: LEARNER
I20260807 11:30:12.716583 462 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01: attempting to promote NON_VOTER 53f447304fdb45bf9a9f7ddf07bf91d3 to VOTER
I20260807 11:30:12.717415 462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } }
I20260807 11:30:12.720103 316 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 LEARNER]: Refusing update from remote peer 7182993a35ce4f5aa488a3caa7acea01: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 11:30:12.720844 32727 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 11:30:12.724344 316 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 53f447304fdb45bf9a9f7ddf07bf91d3 (127.24.103.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:12.727236 32727 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 53f447304fdb45bf9a9f7ddf07bf91d3 (127.24.103.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:12.732636 345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:12.732916 345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:30:12.733783 462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:12.733637 465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:12.734062 465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:12.734079 462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:12.748373 32728 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:12.748922 32728 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:12.749202 32728 tablet_replica.cc:330] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: stopping tablet replica
W20260807 11:30:12.754616 379 connection.cc:441] server connection from 127.0.0.1:55108 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55108 (request call id 4) could send its response
I20260807 11:30:12.757134 32723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.756400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36941 (local address 127.0.0.1:55120)
0807 11:30:12.756808 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:12.756949 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260807 11:30:12.757423 32728 master.cc:552] Master@127.24.103.124:36941 shutdown complete.
I20260807 11:30:12.757555 32728 master_runner.cc:305] Clearing existing system tablet
W20260807 11:30:12.757476 32085 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 -> Peer 53489d42e48a489eb63b33109c42b145 (127.24.103.124:36941): Couldn't send request to peer 53489d42e48a489eb63b33109c42b145. Status: Network error: Client connection negotiation failed: client connection to 127.24.103.124:36941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 11:30:12.758076 32728 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:12.758200 32728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:12.761353 32728 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 11:30:12.763095 469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:12.763473 32728 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:12.763757 32728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 11:30:12.763857 32728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/wal
uuid: "53489d42e48a489eb63b33109c42b145"
format_stamp: "Formatted at 2026-08-07 11:30:12 on dist-test-slave-wl2h"
server_key: "3c403a1dd363d0b86811972f30d30fa5"
server_key_iv: "41dcffe97c4c83e93ee0781f7561ef38"
server_key_version: "encryptionkey@0"
I20260807 11:30:12.769048 32728 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:12.777061 32728 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 11:30:12.777263 32728 log.cc:1196] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 11:30:12.777760 32728 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: Deleting consensus metadata
I20260807 11:30:12.778054 32728 master_runner.cc:315] Copying system tablet from 127.24.103.126:42143
I20260807 11:30:12.780819 32728 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: tablet copy: Beginning tablet copy session from remote peer at address 127.24.103.126:42143
I20260807 11:30:12.798209 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.781651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:35478)
0807 11:30:12.782366 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:12.782479 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:12.782691 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:12.783122 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:12.783127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:12.783707 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:12.784001 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.784014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.785504 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.785510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.787122 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:12.787156 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.788172 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.788179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:12.788374 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.789143 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:12.789167 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:12.790310 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:12.794817 (+ 4507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.794822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.794827 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.795417 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.795905 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:12.795910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:12.795914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:12.796099 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:12.796895 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:12.796902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:12.797122 (+ 220us) client_negotiation.cc:769] Sending connection context
0807 11:30:12.797301 (+ 179us) client_negotiation.cc:240] Negotiation successful
0807 11:30:12.797670 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":435,"threads_started":1}
I20260807 11:30:12.799391 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:12.781730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35478 (local address 127.24.103.126:42143)
0807 11:30:12.781972 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:12.781979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:12.782597 (+ 618us) server_negotiation.cc:407] Connection header received
0807 11:30:12.782853 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:12.782859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:12.782944 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:12.783093 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:12.784175 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.785346 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.787287 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:12.787999 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:12.788126 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:12.790511 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:12.790518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:12.790523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:12.790566 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:12.794624 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.795567 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.795574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.795578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.795775 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:12.796223 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:12.796230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:12.796234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:12.796643 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:12.796813 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:12.798629 (+ 1816us) server_negotiation.cc:299] Negotiation successful
0807 11:30:12.798906 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260807 11:30:12.800015 32126 tablet_copy_service.cc:140] P 7182993a35ce4f5aa488a3caa7acea01: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 53489d42e48a489eb63b33109c42b145 ({username='oryx', principal='oryx/127.24.103.124@KRBTEST.COM'} at 127.0.0.1:35478)
I20260807 11:30:12.800212 32126 tablet_copy_service.cc:161] P 7182993a35ce4f5aa488a3caa7acea01: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 53489d42e48a489eb63b33109c42b145 at {username='oryx', principal='oryx/127.24.103.124@KRBTEST.COM'} at 127.0.0.1:35478: session id = 53489d42e48a489eb63b33109c42b145-00000000000000000000000000000000
I20260807 11:30:12.804390 32126 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 11:30:12.806557 32728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 11:30:12.813772 32728 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: tablet copy: Starting download of 0 data blocks...
I20260807 11:30:12.814116 32728 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: tablet copy: Starting download of 1 WAL segments...
I20260807 11:30:12.817166 32728 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 11:30:12.822470 32126 tablet_copy_service.cc:342] P 7182993a35ce4f5aa488a3caa7acea01: Request end of tablet copy session 53489d42e48a489eb63b33109c42b145-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.103.124@KRBTEST.COM'} at 127.0.0.1:35478
I20260807 11:30:12.822708 32126 tablet_copy_service.cc:433] P 7182993a35ce4f5aa488a3caa7acea01: ending tablet copy session 53489d42e48a489eb63b33109c42b145-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 53489d42e48a489eb63b33109c42b145
I20260807 11:30:12.835232 32728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:12.838384 478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:12.839124 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 32172
W20260807 11:30:12.845099 479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:12.848733 481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:12.850379 32728 server_base.cc:1034] running on GCE node
I20260807 11:30:12.850757 32728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:12.851517 32728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:12.852885 32728 hybrid_clock.cc:648] HybridClock initialized: now 1786102212852834 us; error 43 us; skew 500 ppm
I20260807 11:30:12.853168 32728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:12 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102212, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:12.855150 32728 init.cc:377] Logged in from keytab as oryx/127.24.103.124@KRBTEST.COM (short username oryx)
I20260807 11:30:12.860046 32728 webserver.cc:460] Webserver started at http://127.24.103.124:39711/ using document root <none> and password file <none>
I20260807 11:30:12.860422 32728 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:12.860538 32728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:12.863104 32728 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 11:30:12.864830 489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:12.865254 32728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260807 11:30:12.865394 32728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/wal
uuid: "53489d42e48a489eb63b33109c42b145"
format_stamp: "Formatted at 2026-08-07 11:30:12 on dist-test-slave-wl2h"
server_key: "3c403a1dd363d0b86811972f30d30fa5"
server_key_iv: "41dcffe97c4c83e93ee0781f7561ef38"
server_key_version: "encryptionkey@0"
I20260807 11:30:12.865597 32728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:12.872052 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:46611
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:42143
--webserver_port=43573
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.103.126:42143,127.24.103.125:35933
--tserver_master_addrs=127.24.103.126:42143,127.24.103.125:35933,127.24.103.124:36941 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:30:12.873957 32728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:12.877493 32728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:12.902011 32728 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.124:36941
I20260807 11:30:12.902035 545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.124:36941 every 8 connection(s)
I20260807 11:30:12.902981 32728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
I20260807 11:30:12.910543 546 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:30:12.915920 546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: Bootstrap starting.
I20260807 11:30:12.930755 546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:30:12.931310 546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: Bootstrap complete.
I20260807 11:30:12.932255 546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } }
I20260807 11:30:12.932512 546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 53489d42e48a489eb63b33109c42b145, State: Initialized, Role: LEARNER
I20260807 11:30:12.932781 546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } }
I20260807 11:30:12.933979 549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:12.934324 546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:12.934587 549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: This master's current role is: LEARNER
I20260807 11:30:12.943799 561 catalog_manager.cc:1259] Loaded cluster ID: 13e0d15ed169414d8d7b9cb933658be1
I20260807 11:30:12.943910 561 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: loading cluster ID for follower catalog manager: success
I20260807 11:30:12.946362 561 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: acquiring CA information for follower catalog manager: success
I20260807 11:30:12.948261 561 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:30:13.243669 32723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:13.235969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36941 (local address 127.0.0.1:55126)
0807 11:30:13.236157 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:13.236185 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:13.236300 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:13.237393 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:13.237401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:13.237444 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:13.237683 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:13.237698 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.239332 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.239340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:13.241267 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:13.241287 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.242617 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.242625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:13.242854 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:13.242902 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 11:30:13.243043 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 11:30:13.243253 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 11:30:13.243775 562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:13.236122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.24.103.124:36941)
0807 11:30:13.236852 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:13.236859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:13.236881 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:30:13.236960 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:13.236966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:13.237087 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:13.237331 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:13.237900 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.239165 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.241472 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.242404 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.242581 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:13.242737 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:13.243168 (+ 431us) server_negotiation.cc:299] Negotiation successful
0807 11:30:13.243356 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":360,"threads_started":1}
I20260807 11:30:13.244809 520 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260807 11:30:13.245391 549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7182993a35ce4f5aa488a3caa7acea01. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:13.245697 549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: This master's current role is: LEARNER
W20260807 11:30:13.273569 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:13.273900 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:13.274070 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:13.281451 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:13.281610 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:13.281787 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:30:13.291800 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:46611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=43573
--tserver_master_addrs=127.24.103.126:42143,127.24.103.125:35933,127.24.103.124:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:13.293285 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:13.294665 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:13.308145 567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:13.308241 568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:13.308444 571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:13.309207 501 server_base.cc:1034] running on GCE node
I20260807 11:30:13.309827 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:13.310779 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:13.312029 501 hybrid_clock.cc:648] HybridClock initialized: now 1786102213311938 us; error 63 us; skew 500 ppm
I20260807 11:30:13.312369 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:13 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102213, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:13.315694 501 init.cc:377] Logged in from keytab as oryx/127.24.103.65@KRBTEST.COM (short username oryx)
I20260807 11:30:13.317564 501 webserver.cc:460] Webserver started at http://127.24.103.65:43573/ using document root <none> and password file <none>
I20260807 11:30:13.318292 501 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:13.318459 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:13.324417 501 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 11:30:13.327525 581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:13.328949 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:30:13.329182 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "7422a6ab063542509f75e3098e0dca05"
format_stamp: "Formatted at 2026-08-07 11:30:10 on dist-test-slave-wl2h"
server_key: "ae926a7e318fa1185bcacd1c057078c5"
server_key_iv: "48efe2c0286950adf2badfea89c185be"
server_key_version: "encryptionkey@0"
I20260807 11:30:13.330230 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:13.353607 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:13.357857 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:13.358213 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:13.359462 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:13.360906 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:13.361037 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:13.361174 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:13.361268 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:13.403767 501 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:46611
I20260807 11:30:13.403852 708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:46611 every 8 connection(s)
I20260807 11:30:13.405233 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:30:13.407795 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 501
I20260807 11:30:13.408228 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 32321
Aug 07 11:30:13 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102213, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
Aug 07 11:30:13 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102213, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
Aug 07 11:30:13 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.66@KRBTEST.COM
Aug 07 11:30:13 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102213, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
I20260807 11:30:13.445266 562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:13.408815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:38077 (local address 127.24.103.124:36941)
0807 11:30:13.409085 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:13.409092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:13.421509 (+ 12417us) server_negotiation.cc:407] Connection header received
0807 11:30:13.424514 (+ 3005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:13.424523 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:13.424628 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:13.424744 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:13.426099 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.427258 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.429711 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.430365 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.430460 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:13.437002 (+ 6542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:13.437008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:13.437012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:13.437055 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:13.440979 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:13.442151 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:13.442156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:13.442160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:13.442328 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:13.442904 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:13.442909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:13.442913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:13.443302 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:13.443458 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:13.444594 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0807 11:30:13.444898 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260807 11:30:13.447830 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:13.410651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36941 (local address 127.24.103.65:38077)
0807 11:30:13.421333 (+ 10682us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:13.421359 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:13.421531 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:13.424938 (+ 3407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:13.424943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:13.425610 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:13.425911 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:13.425923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.427456 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.427462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:13.429536 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:13.429551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.430811 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.430818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:13.431163 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:13.432236 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:13.432313 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:13.436701 (+ 4388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:13.441156 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:13.441166 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:13.441183 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:13.441997 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:13.442495 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:13.442502 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:13.442507 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:13.442769 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:13.443622 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:13.443632 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:13.444110 (+ 478us) client_negotiation.cc:769] Sending connection context
0807 11:30:13.445626 (+ 1516us) client_negotiation.cc:240] Negotiation successful
0807 11:30:13.446265 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10509,"mutex_wait_us":111,"spinlock_wait_cycles":32000,"thread_start_us":12606,"threads_started":1}
I20260807 11:30:13.448417 716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:13.409624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.24.103.66:34573)
0807 11:30:13.415608 (+ 5984us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:13.415615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:13.425240 (+ 9625us) server_negotiation.cc:407] Connection header received
0807 11:30:13.425442 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:13.425454 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:13.425731 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:13.425892 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:13.428123 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.429709 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.431621 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.432587 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.432722 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:13.439561 (+ 6839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:13.439576 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:13.439596 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:13.439644 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:13.443738 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:13.444762 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:13.444772 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:13.444778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:13.444944 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:13.445410 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:13.445415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:13.445418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:13.445831 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:13.446495 (+ 664us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:13.447672 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0807 11:30:13.447956 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5809,"thread_start_us":5631,"threads_started":1}
I20260807 11:30:13.448588 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:13.415534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:35933 (local address 127.24.103.65:53953)
0807 11:30:13.423629 (+ 8095us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:13.423645 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:13.423736 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:13.432610 (+ 8874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:13.432616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:13.433315 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:13.433787 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:13.433807 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.435504 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.435510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:13.437255 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:13.437272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.438392 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.438401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:13.438613 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:13.439346 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:13.439363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:13.441647 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:13.445767 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:13.445773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:13.445777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:13.446316 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:13.446724 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:13.446730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:13.446734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:13.446895 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:13.447535 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:13.447540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:13.447709 (+ 169us) client_negotiation.cc:769] Sending connection context
0807 11:30:13.447860 (+ 151us) client_negotiation.cc:240] Negotiation successful
0807 11:30:13.448166 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7969,"thread_start_us":415,"threads_started":1}
I20260807 11:30:13.449571 710 heartbeater.cc:347] Connected to a master server at 127.24.103.124:36941
I20260807 11:30:13.449653 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:13.416183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:53953 (local address 127.24.103.125:35933)
0807 11:30:13.432111 (+ 15928us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:13.432120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:13.432149 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:30:13.432212 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:13.432217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:13.432309 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:13.432811 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:13.434070 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.435330 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.437455 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.438173 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.438973 (+ 800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:13.441892 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:13.441899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:13.441903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:13.441945 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:13.445622 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:13.446471 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:13.446475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:13.446479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:13.446611 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:13.447043 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:13.447047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:13.447050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:13.447403 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:13.448925 (+ 1522us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:13.449047 (+ 122us) server_negotiation.cc:299] Negotiation successful
0807 11:30:13.449283 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15735,"thread_start_us":16222,"threads_started":1}
I20260807 11:30:13.449918 710 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:13.449960 32460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:13.450625 713 heartbeater.cc:347] Connected to a master server at 127.24.103.125:35933
I20260807 11:30:13.450778 713 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:13.450996 710 heartbeater.cc:511] Master 127.24.103.124:36941 requested a full tablet report, sending...
I20260807 11:30:13.451356 713 heartbeater.cc:511] Master 127.24.103.125:35933 requested a full tablet report, sending...
I20260807 11:30:13.453507 305 ts_manager.cc:194] Registered new tserver with Master: 7422a6ab063542509f75e3098e0dca05 (127.24.103.65:46611)
I20260807 11:30:13.453986 510 ts_manager.cc:194] Registered new tserver with Master: 7422a6ab063542509f75e3098e0dca05 (127.24.103.65:46611)
I20260807 11:30:13.464823 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:13.409481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.65:33335)
0807 11:30:13.416482 (+ 7001us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:13.416548 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:13.420180 (+ 3632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:13.422841 (+ 2661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:13.422858 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:13.424016 (+ 1158us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:13.425076 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:13.425106 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.426838 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.426847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:13.429630 (+ 2783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:13.429651 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.431302 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.431311 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:13.431577 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:13.432220 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:13.432252 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:13.435759 (+ 3507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:13.442210 (+ 6451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:13.442215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:13.442218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:13.442736 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:13.443317 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:13.443324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:13.443329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:13.443502 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:13.463646 (+ 20144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:13.463652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:13.463847 (+ 195us) client_negotiation.cc:769] Sending connection context
0807 11:30:13.464002 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 11:30:13.464379 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6613,"mutex_wait_us":9,"thread_start_us":11347,"threads_started":1}
I20260807 11:30:13.464952 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:13.410197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:33335 (local address 127.24.103.126:42143)
0807 11:30:13.422040 (+ 11843us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:13.422050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:13.422076 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:30:13.422172 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:13.422181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:13.422288 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:13.422439 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:13.425371 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.426593 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.429900 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:13.430828 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:13.430976 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:13.436142 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:13.436149 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:13.436154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:13.436196 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:13.442035 (+ 5839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:13.442889 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:13.442896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:13.442901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:13.443107 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:13.443659 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:13.463069 (+ 19410us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:13.463074 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:13.463459 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:13.463674 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:13.464234 (+ 560us) server_negotiation.cc:299] Negotiation successful
0807 11:30:13.464534 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11675,"thread_start_us":11574,"threads_started":1}
I20260807 11:30:13.466325 711 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42143
I20260807 11:30:13.466513 711 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:13.466933 711 heartbeater.cc:511] Master 127.24.103.126:42143 requested a full tablet report, sending...
I20260807 11:30:13.468436 32100 ts_manager.cc:194] Re-registered known tserver with Master: 7422a6ab063542509f75e3098e0dca05 (127.24.103.65:46611)
I20260807 11:30:13.469784 32100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:33335
I20260807 11:30:13.543157 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 32321
I20260807 11:30:13.565454 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:34573
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:42143
--webserver_port=42891
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.103.126:42143,127.24.103.125:35933
--tserver_master_addrs=127.24.103.126:42143,127.24.103.125:35933,127.24.103.124:36941 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:30:13.732085 462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: NON_VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: true } } }
I20260807 11:30:13.732564 462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:13.734871 725 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01: attempting to promote NON_VOTER 53489d42e48a489eb63b33109c42b145 to VOTER
I20260807 11:30:13.735715 725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } }
I20260807 11:30:13.738307 316 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 FOLLOWER]: Refusing update from remote peer 7182993a35ce4f5aa488a3caa7acea01: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 11:30:13.738303 520 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 LEARNER]: Refusing update from remote peer 7182993a35ce4f5aa488a3caa7acea01: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 11:30:13.739059 725 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 11:30:13.739437 462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 11:30:13.743283 462 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 53489d42e48a489eb63b33109c42b145 (127.24.103.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:13.744086 520 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 53489d42e48a489eb63b33109c42b145 (127.24.103.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:13.745144 316 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 53489d42e48a489eb63b33109c42b145 (127.24.103.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:13.747792 724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:13.748073 724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:13.748413 724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:13.748701 724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:13.749037 462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:13.749305 462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:13.750195 730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:13.750473 730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:30:13.751461 731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7182993a35ce4f5aa488a3caa7acea01" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:13.751925 731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 11:30:13.930218 723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:13.930562 723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:13.930697 723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:13.938776 723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:13.938910 723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:13.939116 723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:30:13.949676 723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:34573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=42891
--tserver_master_addrs=127.24.103.126:42143,127.24.103.125:35933,127.24.103.124:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:13.951189 723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:13.952668 723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:13.967749 739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:13.967926 738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:13.967927 741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:13.969412 723 server_base.cc:1034] running on GCE node
I20260807 11:30:13.970103 723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:13.971187 723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:13.972448 723 hybrid_clock.cc:648] HybridClock initialized: now 1786102213972406 us; error 34 us; skew 500 ppm
I20260807 11:30:13.972828 723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:13 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102213, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:13.976522 723 init.cc:377] Logged in from keytab as oryx/127.24.103.66@KRBTEST.COM (short username oryx)
I20260807 11:30:13.978279 723 webserver.cc:460] Webserver started at http://127.24.103.66:42891/ using document root <none> and password file <none>
I20260807 11:30:13.979002 723 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:13.979254 723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:13.984947 723 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:30:13.987946 748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:13.989455 723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:30:13.989670 723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "62c004b483df4454b90ba9594e4625e8"
format_stamp: "Formatted at 2026-08-07 11:30:10 on dist-test-slave-wl2h"
server_key: "fb4bc0aeca468517b5dcc0f28d47680c"
server_key_iv: "842ba04365221918581e1534b59cc568"
server_key_version: "encryptionkey@0"
I20260807 11:30:13.990787 723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:14.022338 723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:14.027269 723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:14.027592 723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:14.028712 723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:14.030272 723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:14.030405 723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:14.030552 723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:14.030650 723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:14.068991 723 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:34573
I20260807 11:30:14.069099 871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:34573 every 8 connection(s)
I20260807 11:30:14.070559 723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:30:14.073942 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 723
I20260807 11:30:14.074266 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 32471
Aug 07 11:30:14 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102213, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
Aug 07 11:30:14 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102213, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
Aug 07 11:30:14 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.67@KRBTEST.COM
Aug 07 11:30:14 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102213, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
I20260807 11:30:14.128737 882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.075567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:45077 (local address 127.24.103.126:42143)
0807 11:30:14.099287 (+ 23720us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.099298 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.100419 (+ 1121us) server_negotiation.cc:407] Connection header received
0807 11:30:14.104254 (+ 3835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.104261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.104374 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.104497 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:14.107912 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.109124 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.113061 (+ 3937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.113515 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.113668 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.119073 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:14.119083 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:14.119088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:14.119148 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:14.123589 (+ 4441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.125411 (+ 1822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.125417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.125421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.125579 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.126448 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.126453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.126457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.126760 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:14.126927 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.127827 (+ 900us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.128091 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23538,"thread_start_us":23752,"threads_started":1}
I20260807 11:30:14.129992 880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.074552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.66:45077)
0807 11:30:14.099764 (+ 25212us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:14.099858 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:14.103828 (+ 3970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:14.105576 (+ 1748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:14.105583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:14.106416 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:14.107684 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.107711 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.110014 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.110018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.112918 (+ 2900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.112934 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.113812 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.113821 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.114253 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.115938 (+ 1685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:14.115967 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:14.118795 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:14.124870 (+ 6075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.124873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.124876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.125270 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.125995 (+ 725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.126001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.126006 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.126319 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.126902 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:14.126910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:14.127423 (+ 513us) client_negotiation.cc:769] Sending connection context
0807 11:30:14.127698 (+ 275us) client_negotiation.cc:240] Negotiation successful
0807 11:30:14.128369 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24710,"mutex_wait_us":460,"spinlock_wait_cycles":4480,"thread_start_us":26372,"threads_started":1}
I20260807 11:30:14.130767 877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.074553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:35933 (local address 127.24.103.66:48931)
0807 11:30:14.101489 (+ 26936us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:14.101520 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:14.103830 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:14.104960 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:14.104967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:14.106378 (+ 1411us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:14.107696 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.107713 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.110199 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.110209 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.112893 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.112903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.113812 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.113821 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.114244 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.115182 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:14.115216 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:14.118655 (+ 3439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:14.124411 (+ 5756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.124419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.124433 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.124982 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.126023 (+ 1041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.126313 (+ 290us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.126318 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.127128 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.129554 (+ 2426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:14.129561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:14.129749 (+ 188us) client_negotiation.cc:769] Sending connection context
0807 11:30:14.129923 (+ 174us) client_negotiation.cc:240] Negotiation successful
0807 11:30:14.130246 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26766,"mutex_wait_us":1262,"spinlock_wait_cycles":54144,"thread_start_us":26358,"threads_started":1}
I20260807 11:30:14.131371 878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.074975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:48931 (local address 127.24.103.125:35933)
0807 11:30:14.098840 (+ 23865us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.098852 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.101626 (+ 2774us) server_negotiation.cc:407] Connection header received
0807 11:30:14.104299 (+ 2673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.104307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.104402 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.104503 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:14.107913 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.108875 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.113029 (+ 4154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.113522 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.113673 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.119222 (+ 5549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:14.119232 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:14.119238 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:14.119301 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:14.124242 (+ 4941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.125721 (+ 1479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.125728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.125732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.125897 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.128602 (+ 2705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.128610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.128615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.129423 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:14.129607 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.130558 (+ 951us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.131089 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23631,"thread_start_us":26963,"threads_started":1}
I20260807 11:30:14.131685 873 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42143
I20260807 11:30:14.132066 873 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:14.132135 883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.075554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.24.103.67:37329)
0807 11:30:14.103408 (+ 27854us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.103415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.103441 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:30:14.103500 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.103506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.103772 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.105182 (+ 1410us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:14.106687 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.108558 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.110114 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.111116 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.111258 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.120580 (+ 9322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:14.120593 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:14.120606 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:14.120647 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:14.124728 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.125693 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.125702 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.125706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.125857 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.126466 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.126472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.126477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.126953 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:14.127314 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.129801 (+ 2487us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.131627 (+ 1826us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27688,"thread_start_us":27516,"threads_started":1}
I20260807 11:30:14.132321 874 heartbeater.cc:347] Connected to a master server at 127.24.103.125:35933
I20260807 11:30:14.132477 874 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:14.132802 876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.073789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:35501 (local address 127.24.103.124:36941)
0807 11:30:14.101535 (+ 27746us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.101542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.102789 (+ 1247us) server_negotiation.cc:407] Connection header received
0807 11:30:14.104382 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.104387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.104481 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.104604 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:14.108879 (+ 4275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.109844 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.114335 (+ 4491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.114996 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.115167 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.123264 (+ 8097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:14.123271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:14.123277 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:14.123323 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:14.128056 (+ 4733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.128975 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.128981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.128984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.129142 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.129613 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.129619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.129622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.130013 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:14.130173 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.132139 (+ 1966us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.132378 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27618,"thread_start_us":27499,"threads_started":1}
I20260807 11:30:14.133188 873 heartbeater.cc:511] Master 127.24.103.126:42143 requested a full tablet report, sending...
I20260807 11:30:14.133210 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.076019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36941 (local address 127.24.103.66:35501)
0807 11:30:14.102630 (+ 26611us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:14.102662 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:14.104213 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:14.104956 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:14.104965 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:14.106377 (+ 1412us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:14.107689 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.107714 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.110023 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.110030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.112877 (+ 2847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.112894 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.115281 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.115288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.115513 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.117566 (+ 2053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:14.117595 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:14.122970 (+ 5375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:14.128236 (+ 5266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.128243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.128247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.128807 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.129278 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.129284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.129288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.129456 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.130245 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:14.130252 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:14.130409 (+ 157us) client_negotiation.cc:769] Sending connection context
0807 11:30:14.130894 (+ 485us) client_negotiation.cc:240] Negotiation successful
0807 11:30:14.131234 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26432,"mutex_wait_us":407,"spinlock_wait_cycles":36864,"thread_start_us":26191,"threads_started":1}
I20260807 11:30:14.134246 872 heartbeater.cc:347] Connected to a master server at 127.24.103.124:36941
I20260807 11:30:14.134407 872 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:14.134734 32607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:14.134893 872 heartbeater.cc:511] Master 127.24.103.124:36941 requested a full tablet report, sending...
I20260807 11:30:14.135926 32100 ts_manager.cc:194] Re-registered known tserver with Master: 62c004b483df4454b90ba9594e4625e8 (127.24.103.66:34573)
I20260807 11:30:14.136129 510 ts_manager.cc:194] Registered new tserver with Master: 62c004b483df4454b90ba9594e4625e8 (127.24.103.66:34573)
I20260807 11:30:14.137255 32100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:45077
I20260807 11:30:14.137853 874 heartbeater.cc:511] Master 127.24.103.125:35933 requested a full tablet report, sending...
I20260807 11:30:14.139199 305 ts_manager.cc:194] Registered new tserver with Master: 62c004b483df4454b90ba9594e4625e8 (127.24.103.66:34573)
I20260807 11:30:14.230459 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 32471
I20260807 11:30:14.251957 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:37329
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:42143
--webserver_port=44735
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.103.126:42143,127.24.103.125:35933
--tserver_master_addrs=127.24.103.126:42143,127.24.103.125:35933,127.24.103.124:36941 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:30:14.472522 711 heartbeater.cc:503] Master 127.24.103.126:42143 was elected leader, sending a full tablet report...
W20260807 11:30:14.601817 886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:14.602134 886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:14.602289 886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:14.609987 886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:14.610140 886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:14.610304 886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:30:14.620621 886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:37329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=44735
--tserver_master_addrs=127.24.103.126:42143,127.24.103.125:35933,127.24.103.124:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:14.622200 886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:14.623586 886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:14.637755 891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:14.637836 892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:14.637753 895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:14.639555 886 server_base.cc:1034] running on GCE node
I20260807 11:30:14.640249 886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:14.641525 886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:14.642748 886 hybrid_clock.cc:648] HybridClock initialized: now 1786102214642704 us; error 42 us; skew 500 ppm
I20260807 11:30:14.643142 886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:14 dist-test-slave-wl2h krb5kdc[32053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102214, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:14.646572 886 init.cc:377] Logged in from keytab as oryx/127.24.103.67@KRBTEST.COM (short username oryx)
I20260807 11:30:14.648207 886 webserver.cc:460] Webserver started at http://127.24.103.67:44735/ using document root <none> and password file <none>
I20260807 11:30:14.649001 886 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:14.649149 886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:14.654917 886 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:30:14.657956 902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:14.659741 886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:30:14.659930 886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "eae8abbb0c9242f49c9dd17ad13de1a6"
format_stamp: "Formatted at 2026-08-07 11:30:11 on dist-test-slave-wl2h"
server_key: "16f99554a8ae0f74bcb9df6c43f1071b"
server_key_iv: "db758143e5dd7d985254621fc4cd58ba"
server_key_version: "encryptionkey@0"
I20260807 11:30:14.660985 886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:14.693763 886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:14.698591 886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:14.698908 886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:14.700052 886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:14.701517 886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:14.701610 886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:14.701759 886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:14.701861 886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:14.738644 886 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:37329
I20260807 11:30:14.738740 1028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:37329 every 8 connection(s)
I20260807 11:30:14.740214 886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:30:14.740895 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 886
I20260807 11:30:14.805984 1048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.761608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.24.103.126:42143)
0807 11:30:14.774366 (+ 12758us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.774370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.774389 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:30:14.774444 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.774448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.774521 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.774663 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:14.778015 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.778843 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.781881 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.782653 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.782795 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.795167 (+ 12372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:14.795176 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:14.795180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:14.795233 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:14.800394 (+ 5161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.802891 (+ 2497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.802897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.802901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.803107 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.803680 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.803687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.803690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.804074 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:14.804331 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.804971 (+ 640us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.805433 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12632,"thread_start_us":12516,"threads_started":1}
Aug 07 11:30:14 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102214, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
Aug 07 11:30:14 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102214, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.124@KRBTEST.COM
Aug 07 11:30:14 dist-test-slave-wl2h krb5kdc[32053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102214, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.125@KRBTEST.COM
I20260807 11:30:14.817131 1046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.761270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.24.103.125:35933)
0807 11:30:14.782327 (+ 21057us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.782336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.782362 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:30:14.782443 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.782452 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.782583 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.787404 (+ 4821us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:14.787699 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.789479 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.794896 (+ 5417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.795653 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.795797 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.802428 (+ 6631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:14.802434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:14.802438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:14.802485 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:14.813045 (+ 10560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.813957 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.813963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.813967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.814145 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.814601 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.814606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.814610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.814975 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:14.816295 (+ 1320us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.816425 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.816690 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20925,"thread_start_us":20835,"threads_started":1}
I20260807 11:30:14.817203 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.771186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.24.103.124:36941)
0807 11:30:14.771821 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.771830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.773754 (+ 1924us) server_negotiation.cc:407] Connection header received
0807 11:30:14.774006 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.774024 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.774132 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.774257 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:14.777356 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.779193 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.782362 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.783147 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.783339 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.795245 (+ 11906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:14.795252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:14.795256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:14.795302 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:14.804046 (+ 8744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.808858 (+ 4812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.808865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.808869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.809109 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.810850 (+ 1741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.810855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.810858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.811360 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:14.811683 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.816400 (+ 4717us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.816713 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":311,"threads_started":1}
I20260807 11:30:14.826480 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.743367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:38971 (local address 127.24.103.124:36941)
0807 11:30:14.756264 (+ 12897us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.756274 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.783719 (+ 27445us) server_negotiation.cc:407] Connection header received
0807 11:30:14.797012 (+ 13293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.797018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.797381 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.797504 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:14.800547 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.801942 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.808084 (+ 6142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.808790 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.808900 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.817230 (+ 8330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:14.817239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:14.817246 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:14.817291 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:14.821908 (+ 4617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.823381 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.823386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.823389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.823635 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.824410 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.824415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.824418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.824780 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:14.824904 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.825777 (+ 873us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.826030 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12713,"thread_start_us":12603,"threads_started":1}
I20260807 11:30:14.826515 1034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.743707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:33959 (local address 127.24.103.126:42143)
0807 11:30:14.755836 (+ 12129us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.755846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.794367 (+ 38521us) server_negotiation.cc:407] Connection header received
0807 11:30:14.796975 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.796982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.797077 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.797222 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:14.800189 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.801520 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.807463 (+ 5943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.808235 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.808372 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.817833 (+ 9461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:14.817841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:14.817846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:14.817898 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:14.821908 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.823260 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.823267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.823272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.823451 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.824067 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.824073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.824078 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.824529 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:14.824679 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.825787 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.826030 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11953,"thread_start_us":11841,"threads_started":1}
I20260807 11:30:14.813598 32100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:30:14.828949 32100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:30:14.829083 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.749924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:35933 (local address 127.24.103.67:47521)
0807 11:30:14.801921 (+ 51997us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:14.801944 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:14.802171 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:14.803422 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:14.803427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:14.804057 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:14.804412 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.804425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.806402 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.806408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.808033 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.808051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.809358 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.809365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.809996 (+ 631us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.813523 (+ 3527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:14.813552 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:14.819741 (+ 6189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:14.825026 (+ 5285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.825031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.825035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.825640 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.826116 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.826122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.826126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.826318 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.827934 (+ 1616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:14.827938 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:14.828102 (+ 164us) client_negotiation.cc:769] Sending connection context
0807 11:30:14.828292 (+ 190us) client_negotiation.cc:240] Negotiation successful
0807 11:30:14.828577 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51829,"mutex_wait_us":440,"thread_start_us":33039,"threads_started":1}
I20260807 11:30:14.829206 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.744099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36941 (local address 127.24.103.67:38971)
0807 11:30:14.783530 (+ 39431us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:14.783593 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:14.796634 (+ 13041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:14.797863 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:14.797877 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:14.798848 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:14.800314 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.800328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.804652 (+ 4324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.804658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.807347 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.807365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.809541 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.809547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.810258 (+ 711us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.811909 (+ 1651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:14.811932 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:14.816856 (+ 4924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:14.822100 (+ 5244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.822111 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.822133 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.823182 (+ 1049us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.823996 (+ 814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.824033 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.824041 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.824279 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.824973 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:14.824977 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:14.825283 (+ 306us) client_negotiation.cc:769] Sending connection context
0807 11:30:14.826827 (+ 1544us) client_negotiation.cc:240] Negotiation successful
0807 11:30:14.827530 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39048,"mutex_wait_us":510,"spinlock_wait_cycles":896,"thread_start_us":57416,"threads_started":1}
I20260807 11:30:14.830013 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.744099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.67:33959)
0807 11:30:14.794218 (+ 50119us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:14.794254 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:14.796634 (+ 2380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:14.797863 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:14.797877 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:14.799446 (+ 1569us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:14.799947 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.799979 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.802851 (+ 2872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.802858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.807265 (+ 4407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.807324 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.809358 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.809365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.809842 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.810924 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:14.810972 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:14.817542 (+ 6570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:14.822100 (+ 4558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.822111 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.822133 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.823096 (+ 963us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.823620 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:14.823625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:14.823628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:14.823928 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:14.824782 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:14.824791 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:14.825283 (+ 492us) client_negotiation.cc:769] Sending connection context
0807 11:30:14.826751 (+ 1468us) client_negotiation.cc:240] Negotiation successful
0807 11:30:14.827640 (+ 889us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49778,"mutex_wait_us":374,"spinlock_wait_cycles":84224,"thread_start_us":49459,"threads_started":1}
I20260807 11:30:14.829167 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.750463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:47521 (local address 127.24.103.125:35933)
0807 11:30:14.781986 (+ 31523us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.781995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.802249 (+ 20254us) server_negotiation.cc:407] Connection header received
0807 11:30:14.802329 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.802334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.802420 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.802524 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:14.804618 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.806133 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.808275 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.808991 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.809120 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.820832 (+ 11712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:14.820839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:14.820843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:14.820891 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:14.824845 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.825802 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.825808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.825811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.825978 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:14.826467 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:14.826473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:14.826476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:14.827282 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:14.827702 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.828461 (+ 759us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.828696 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31342,"thread_start_us":31187,"threads_started":1}
I20260807 11:30:14.830893 1029 heartbeater.cc:347] Connected to a master server at 127.24.103.124:36941
I20260807 11:30:14.831251 1029 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:14.831485 1031 heartbeater.cc:347] Connected to a master server at 127.24.103.125:35933
I20260807 11:30:14.831647 1031 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:14.832228 1029 heartbeater.cc:511] Master 127.24.103.124:36941 requested a full tablet report, sending...
I20260807 11:30:14.832264 1031 heartbeater.cc:511] Master 127.24.103.125:35933 requested a full tablet report, sending...
I20260807 11:30:14.832536 1030 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42143
I20260807 11:30:14.832674 1030 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:14.833237 1030 heartbeater.cc:511] Master 127.24.103.126:42143 requested a full tablet report, sending...
I20260807 11:30:14.834234 305 ts_manager.cc:194] Registered new tserver with Master: eae8abbb0c9242f49c9dd17ad13de1a6 (127.24.103.67:37329)
I20260807 11:30:14.834291 510 ts_manager.cc:194] Registered new tserver with Master: eae8abbb0c9242f49c9dd17ad13de1a6 (127.24.103.67:37329)
I20260807 11:30:14.835331 32101 ts_manager.cc:194] Re-registered known tserver with Master: eae8abbb0c9242f49c9dd17ad13de1a6 (127.24.103.67:37329)
I20260807 11:30:14.836284 32101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:33959
I20260807 11:30:14.873597 1067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.859853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37329 (local address 127.0.0.1:49366)
0807 11:30:14.864295 (+ 4442us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:14.864319 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:14.864544 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:14.865901 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:14.865906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:14.865936 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:14.866538 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.866553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.868570 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.868577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.870423 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.870435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.872768 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.872773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.872903 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.872927 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 11:30:14.873005 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 11:30:14.873105 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4216,"mutex_wait_us":1,"thread_start_us":2341,"threads_started":1}
I20260807 11:30:14.873669 1068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.860748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.24.103.67:37329)
0807 11:30:14.865310 (+ 4562us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.865318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.865356 (+ 38us) server_negotiation.cc:407] Connection header received
0807 11:30:14.865456 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.865463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.865662 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.865867 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:14.866801 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.868327 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.870597 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.872557 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.872707 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.872904 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.873069 (+ 165us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.873272 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4321,"thread_start_us":3965,"threads_started":1}
I20260807 11:30:14.874282 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.859301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:46611 (local address 127.0.0.1:48514)
0807 11:30:14.861734 (+ 2433us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:14.861759 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:14.861915 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:14.863838 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:14.863841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:14.863867 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:14.864281 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.864300 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.866869 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.866874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.869455 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.869468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.871443 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.871450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.873351 (+ 1901us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.873396 (+ 45us) client_negotiation.cc:769] Sending connection context
0807 11:30:14.873487 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 11:30:14.873631 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2104,"mutex_wait_us":1745,"thread_start_us":1985,"threads_started":1}
I20260807 11:30:14.875211 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.859578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48514 (local address 127.24.103.65:46611)
0807 11:30:14.862973 (+ 3395us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.862988 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.863047 (+ 59us) server_negotiation.cc:407] Connection header received
0807 11:30:14.863181 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.863194 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.863560 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.863841 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:14.864654 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.866702 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.869633 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.871208 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.871365 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.871610 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.874639 (+ 3029us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.874845 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3140,"thread_start_us":3013,"threads_started":1}
I20260807 11:30:14.876340 1070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.862600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34573 (local address 127.0.0.1:35042)
0807 11:30:14.866740 (+ 4140us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:14.866760 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:14.866883 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:14.868501 (+ 1618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:14.868510 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:14.868539 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:14.869085 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.869099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.870911 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.870916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.873145 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.873160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.874986 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.874993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.875224 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.875247 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 11:30:14.875357 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 11:30:14.875597 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4007,"mutex_wait_us":1531,"thread_start_us":2242,"threads_started":1}
I20260807 11:30:14.877110 1069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.862834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.24.103.66:34573)
0807 11:30:14.863385 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.863398 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.867902 (+ 4504us) server_negotiation.cc:407] Connection header received
0807 11:30:14.868020 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.868030 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.868284 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.868489 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:14.869298 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.870753 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.873317 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.874746 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.874914 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.875168 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.876616 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.876820 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":318,"threads_started":1}
I20260807 11:30:14.878249 957 tablet_service.cc:1467] Processing CreateTablet for tablet 45cf06d924e0446d921c74e711cd7543 (DEFAULT_TABLE table=test-table [id=f18e3418b0584a75931b8cae57fa98d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:14.878697 636 tablet_service.cc:1467] Processing CreateTablet for tablet 45cf06d924e0446d921c74e711cd7543 (DEFAULT_TABLE table=test-table [id=f18e3418b0584a75931b8cae57fa98d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:14.879942 957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cf06d924e0446d921c74e711cd7543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:14.880033 636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cf06d924e0446d921c74e711cd7543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:14.880844 803 tablet_service.cc:1467] Processing CreateTablet for tablet 45cf06d924e0446d921c74e711cd7543 (DEFAULT_TABLE table=test-table [id=f18e3418b0584a75931b8cae57fa98d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:14.882151 803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cf06d924e0446d921c74e711cd7543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:14.896656 1073 tablet_bootstrap.cc:492] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8: Bootstrap starting.
I20260807 11:30:14.901463 1073 tablet_bootstrap.cc:654] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:14.902890 1073 log.cc:824] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:14.907624 1071 tablet_bootstrap.cc:492] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05: Bootstrap starting.
I20260807 11:30:14.910974 1071 tablet_bootstrap.cc:654] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:14.907202 1072 tablet_bootstrap.cc:492] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: Bootstrap starting.
I20260807 11:30:14.912616 1071 log.cc:824] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:14.915728 1072 tablet_bootstrap.cc:654] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:14.917414 1072 log.cc:824] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:14.921705 1071 tablet_bootstrap.cc:492] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05: No bootstrap required, opened a new log
I20260807 11:30:14.921705 1073 tablet_bootstrap.cc:492] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8: No bootstrap required, opened a new log
I20260807 11:30:14.922108 1073 ts_tablet_manager.cc:1397] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.010s
I20260807 11:30:14.922101 1071 ts_tablet_manager.cc:1397] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260807 11:30:14.923173 1072 tablet_bootstrap.cc:492] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: No bootstrap required, opened a new log
I20260807 11:30:14.923467 1072 ts_tablet_manager.cc:1397] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20260807 11:30:14.933576 1071 raft_consensus.cc:348] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } } peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } }
I20260807 11:30:14.933576 1073 raft_consensus.cc:348] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } } peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } }
I20260807 11:30:14.933631 1072 raft_consensus.cc:348] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } } peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } }
I20260807 11:30:14.934108 1073 raft_consensus.cc:374] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:14.934098 1071 raft_consensus.cc:374] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:14.934196 1072 raft_consensus.cc:374] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:14.934208 1073 raft_consensus.cc:729] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c004b483df4454b90ba9594e4625e8, State: Initialized, Role: FOLLOWER
I20260807 11:30:14.934202 1071 raft_consensus.cc:729] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7422a6ab063542509f75e3098e0dca05, State: Initialized, Role: FOLLOWER
I20260807 11:30:14.934322 1072 raft_consensus.cc:729] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae8abbb0c9242f49c9dd17ad13de1a6, State: Initialized, Role: FOLLOWER
I20260807 11:30:14.934989 1071 consensus_queue.cc:260] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } } peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } }
I20260807 11:30:14.934989 1073 consensus_queue.cc:260] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } } peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } }
I20260807 11:30:14.935459 1072 consensus_queue.cc:260] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } } peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } }
I20260807 11:30:14.935859 1077 raft_consensus.cc:482] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:30:14.936280 1077 raft_consensus.cc:504] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } } peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } }
I20260807 11:30:14.939456 1077 leader_election.cc:288] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7422a6ab063542509f75e3098e0dca05 (127.24.103.65:46611), 62c004b483df4454b90ba9594e4625e8 (127.24.103.66:34573)
I20260807 11:30:14.946771 1030 heartbeater.cc:503] Master 127.24.103.126:42143 was elected leader, sending a full tablet report...
I20260807 11:30:14.951097 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.940346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34573 (local address 127.24.103.67:35603)
0807 11:30:14.942191 (+ 1845us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:14.942214 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:14.942330 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:14.942966 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:14.942973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:14.943031 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:14.943238 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.943250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.944886 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.944891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.947357 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.947375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.950127 (+ 2752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.950134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.950372 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.950410 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 11:30:14.950530 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 11:30:14.950723 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1699,"mutex_wait_us":1}
I20260807 11:30:14.951337 1071 ts_tablet_manager.cc:1428] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05: Time spent starting tablet: real 0.029s user 0.013s sys 0.007s
I20260807 11:30:14.951862 1069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.942247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:35603 (local address 127.24.103.66:34573)
0807 11:30:14.942481 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.942488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.942507 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:30:14.942589 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.942593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.942671 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.942774 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:14.943443 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.944666 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.947597 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.949365 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.949557 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.949661 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.951352 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.951509 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:30:14.953019 1073 ts_tablet_manager.cc:1428] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8: Time spent starting tablet: real 0.031s user 0.013s sys 0.014s
I20260807 11:30:14.952908 823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cf06d924e0446d921c74e711cd7543" candidate_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c004b483df4454b90ba9594e4625e8" is_pre_election: true
I20260807 11:30:14.953545 823 raft_consensus.cc:2454] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eae8abbb0c9242f49c9dd17ad13de1a6 in term 0.
I20260807 11:30:14.955158 904 leader_election.cc:302] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c004b483df4454b90ba9594e4625e8, eae8abbb0c9242f49c9dd17ad13de1a6; no voters:
I20260807 11:30:14.956405 873 heartbeater.cc:503] Master 127.24.103.126:42143 was elected leader, sending a full tablet report...
I20260807 11:30:14.957168 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.939811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:46611 (local address 127.24.103.67:38145)
0807 11:30:14.940829 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:14.940857 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:14.941015 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:14.941915 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:14.941920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:14.941946 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:14.942182 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.942199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.944135 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.944143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.949908 (+ 5765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:14.949926 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.956249 (+ 6323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.956255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:14.956503 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.956547 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 11:30:14.956671 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 11:30:14.956828 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":2479}
I20260807 11:30:14.957886 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.941320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:38145 (local address 127.24.103.65:46611)
0807 11:30:14.941555 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.941563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.941586 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:30:14.941677 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.941685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.941785 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.941893 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:14.942401 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.943924 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.954282 (+ 10358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.955836 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.955992 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.956068 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.957450 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.957615 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 11:30:14.958532 1081 raft_consensus.cc:2781] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:30:14.958736 1081 raft_consensus.cc:482] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:30:14.958865 1081 raft_consensus.cc:3037] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:14.958912 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cf06d924e0446d921c74e711cd7543" candidate_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7422a6ab063542509f75e3098e0dca05" is_pre_election: true
I20260807 11:30:14.959501 659 raft_consensus.cc:2454] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eae8abbb0c9242f49c9dd17ad13de1a6 in term 0.
I20260807 11:30:14.962072 1072 ts_tablet_manager.cc:1428] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: Time spent starting tablet: real 0.038s user 0.015s sys 0.005s
I20260807 11:30:14.962739 1081 raft_consensus.cc:504] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } } peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } }
I20260807 11:30:14.963636 1081 leader_election.cc:288] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [CANDIDATE]: Term 1 election: Requested vote from peers 7422a6ab063542509f75e3098e0dca05 (127.24.103.65:46611), 62c004b483df4454b90ba9594e4625e8 (127.24.103.66:34573)
I20260807 11:30:14.964375 823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cf06d924e0446d921c74e711cd7543" candidate_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c004b483df4454b90ba9594e4625e8"
I20260807 11:30:14.964488 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cf06d924e0446d921c74e711cd7543" candidate_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7422a6ab063542509f75e3098e0dca05"
I20260807 11:30:14.964653 823 raft_consensus.cc:3037] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:14.964720 659 raft_consensus.cc:3037] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:14.967962 659 raft_consensus.cc:2454] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eae8abbb0c9242f49c9dd17ad13de1a6 in term 1.
I20260807 11:30:14.968397 905 leader_election.cc:302] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7422a6ab063542509f75e3098e0dca05, eae8abbb0c9242f49c9dd17ad13de1a6; no voters:
I20260807 11:30:14.968636 823 raft_consensus.cc:2454] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eae8abbb0c9242f49c9dd17ad13de1a6 in term 1.
I20260807 11:30:14.968855 1077 raft_consensus.cc:2781] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:14.969224 1077 raft_consensus.cc:686] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 1 LEADER]: Becoming Leader. State: Replica: eae8abbb0c9242f49c9dd17ad13de1a6, State: Running, Role: LEADER
I20260807 11:30:14.969870 1077 consensus_queue.cc:237] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } } peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } }
I20260807 11:30:14.974517 32101 catalog_manager.cc:5495] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eae8abbb0c9242f49c9dd17ad13de1a6 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } health_report { overall_health: UNKNOWN } } }
I20260807 11:30:14.996826 1068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:14.990982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.24.103.67:37329)
0807 11:30:14.991193 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:14.991198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:14.991222 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:30:14.991357 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:14.991360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:14.991440 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:14.991563 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:30:14.992004 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.992769 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.994262 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:14.994655 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:14.994808 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:14.995385 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:30:14.995993 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:30:14.996134 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:14.996309 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 11:30:14.996553 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 11:30:15.003507 1037 tablet.cc:2369] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:30:15.005574 659 raft_consensus.cc:1261] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05 [term 1 FOLLOWER]: Refusing update from remote peer eae8abbb0c9242f49c9dd17ad13de1a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:15.005575 823 raft_consensus.cc:1261] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8 [term 1 FOLLOWER]: Refusing update from remote peer eae8abbb0c9242f49c9dd17ad13de1a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:15.006676 1077 consensus_queue.cc:1035] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:15.006976 1081 consensus_queue.cc:1035] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:15.023415 1087 mvcc.cc:205] Tried to move back new op lower bound from 7315874672652099584 to 7315874672520970240. Current Snapshot: MvccSnapshot[applied={T|T < 7315874672652099584}]
I20260807 11:30:15.023881 1088 mvcc.cc:205] Tried to move back new op lower bound from 7315874672652099584 to 7315874672520970240. Current Snapshot: MvccSnapshot[applied={T|T < 7315874672652099584}]
I20260807 11:30:15.044066 32100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:15.044406 32100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:15.050323 32100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7182993a35ce4f5aa488a3caa7acea01: Sending DeleteTablet for 3 replicas of tablet 45cf06d924e0446d921c74e711cd7543
I20260807 11:30:15.051831 636 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cf06d924e0446d921c74e711cd7543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:15 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:48514
I20260807 11:30:15.052371 957 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cf06d924e0446d921c74e711cd7543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:15 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:49366
I20260807 11:30:15.053011 1097 tablet_replica.cc:330] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05: stopping tablet replica
I20260807 11:30:15.053511 1097 raft_consensus.cc:2229] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:15.055150 803 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cf06d924e0446d921c74e711cd7543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:15 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:35042
I20260807 11:30:15.056118 1098 tablet_replica.cc:330] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: stopping tablet replica
I20260807 11:30:15.056674 1098 raft_consensus.cc:2229] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:30:15.057472 1098 raft_consensus.cc:2258] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:15.060201 1098 ts_tablet_manager.cc:1905] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:15.060401 1097 raft_consensus.cc:2258] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:15.066946 1100 tablet_replica.cc:330] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8: stopping tablet replica
I20260807 11:30:15.067514 1100 raft_consensus.cc:2229] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:15.067673 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 32066
I20260807 11:30:15.068065 1100 raft_consensus.cc:2258] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:15.068854 32136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:15.069999 1098 ts_tablet_manager.cc:1918] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:15.070185 1098 log.cc:1196] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/45cf06d924e0446d921c74e711cd7543
I20260807 11:30:15.070669 1098 ts_tablet_manager.cc:1939] T 45cf06d924e0446d921c74e711cd7543 P eae8abbb0c9242f49c9dd17ad13de1a6: Deleting consensus metadata
I20260807 11:30:15.072280 32085 catalog_manager.cc:4841] TS eae8abbb0c9242f49c9dd17ad13de1a6 (127.24.103.67:37329): tablet 45cf06d924e0446d921c74e711cd7543 (table test-table [id=f18e3418b0584a75931b8cae57fa98d6]) successfully deleted
I20260807 11:30:15.075938 1097 ts_tablet_manager.cc:1905] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:15.076844 1100 ts_tablet_manager.cc:1905] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:15.084702 1100 ts_tablet_manager.cc:1918] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:15.084975 1100 log.cc:1196] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/45cf06d924e0446d921c74e711cd7543
I20260807 11:30:15.085740 1100 ts_tablet_manager.cc:1939] T 45cf06d924e0446d921c74e711cd7543 P 62c004b483df4454b90ba9594e4625e8: Deleting consensus metadata
I20260807 11:30:15.089613 1097 ts_tablet_manager.cc:1918] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:15.089810 1097 log.cc:1196] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/45cf06d924e0446d921c74e711cd7543
I20260807 11:30:15.090428 1097 ts_tablet_manager.cc:1939] T 45cf06d924e0446d921c74e711cd7543 P 7422a6ab063542509f75e3098e0dca05: Deleting consensus metadata
I20260807 11:30:15.100425 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:15.055333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54726 (local address 127.24.103.125:35933)
0807 11:30:15.055538 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:15.055547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:15.067360 (+ 11813us) server_negotiation.cc:407] Connection header received
0807 11:30:15.075593 (+ 8233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:15.075603 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:15.075743 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:15.075880 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:15.079769 (+ 3889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:15.080982 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:15.083152 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:15.083844 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:15.083940 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:15.093325 (+ 9385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:15.093333 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:15.093338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:15.093383 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:15.097094 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:15.098034 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:15.098040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:15.098044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:15.098193 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:15.098654 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:15.098658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:15.098661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:15.099046 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:15.099276 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:15.099851 (+ 575us) server_negotiation.cc:299] Negotiation successful
0807 11:30:15.100089 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 11:30:15.118027 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:15.066494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.24.103.124:36941)
0807 11:30:15.066681 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:15.066688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:15.102695 (+ 36007us) server_negotiation.cc:407] Connection header received
0807 11:30:15.102919 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:15.102925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:15.103038 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:15.103102 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:15.106829 (+ 3727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:15.107811 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:15.109278 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:15.109737 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:15.109870 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:15.112553 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:15.112559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:15.112564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:15.112604 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:15.115381 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:15.116086 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:15.116091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:15.116095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:15.116227 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:15.116582 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:15.116606 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:15.116611 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:15.116893 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:15.117153 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:15.117584 (+ 431us) server_negotiation.cc:299] Negotiation successful
0807 11:30:15.117802 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 11:30:15.239780 32085 catalog_manager.cc:4841] TS 62c004b483df4454b90ba9594e4625e8 (127.24.103.66:34573): tablet 45cf06d924e0446d921c74e711cd7543 (table test-table [id=f18e3418b0584a75931b8cae57fa98d6]) successfully deleted
I20260807 11:30:15.239789 32086 catalog_manager.cc:4841] TS 7422a6ab063542509f75e3098e0dca05 (127.24.103.65:46611): tablet 45cf06d924e0446d921c74e711cd7543 (table test-table [id=f18e3418b0584a75931b8cae57fa98d6]) successfully deleted
I20260807 11:30:15.240433 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 32066
I20260807 11:30:15.297216 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:15.271954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.24.103.125:35933)
0807 11:30:15.272432 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:15.272440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:15.275740 (+ 3300us) server_negotiation.cc:407] Connection header received
0807 11:30:15.276098 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:15.276109 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:15.276235 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:15.276331 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:15.281189 (+ 4858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:15.282478 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:15.284919 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:15.285617 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:15.285756 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:15.289645 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:15.289650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:15.289653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:15.289691 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:15.293287 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:15.294224 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:15.294229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:15.294233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:15.294394 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:15.294851 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:15.294856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:15.294859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:15.295462 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:15.295693 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:15.296567 (+ 874us) server_negotiation.cc:299] Negotiation successful
0807 11:30:15.296822 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318}
I20260807 11:30:15.297495 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:15.272184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.24.103.124:36941)
0807 11:30:15.272656 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:15.272665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:15.280558 (+ 7893us) server_negotiation.cc:407] Connection header received
0807 11:30:15.280738 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:15.280747 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:15.280846 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:15.280953 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:15.283673 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:15.284439 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:15.286244 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:15.286885 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:15.287053 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:15.290278 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:15.290283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:15.290286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:15.290320 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:15.293758 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:15.294678 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:15.294685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:15.294690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:15.294857 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:15.295316 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:15.295323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:15.295326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:15.295728 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:15.295988 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:15.296897 (+ 909us) server_negotiation.cc:299] Negotiation successful
0807 11:30:15.297120 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
I20260807 11:30:16.040273 1110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.037965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.65:43855)
0807 11:30:16.039690 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.039918 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1521,"threads_started":1}
W20260807 11:30:16.041038 711 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:42143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
I20260807 11:30:16.042043 1111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.040066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.66:37451)
0807 11:30:16.040954 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.041737 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":921,"threads_started":1}
W20260807 11:30:16.042614 873 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:42143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
I20260807 11:30:16.044376 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.043157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.67:42891)
0807 11:30:16.043822 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.044026 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":336,"threads_started":1}
W20260807 11:30:16.044777 1030 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:42143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
I20260807 11:30:16.552342 1113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7182993a35ce4f5aa488a3caa7acea01)
I20260807 11:30:16.552783 1113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } }
I20260807 11:30:16.554177 1114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7182993a35ce4f5aa488a3caa7acea01)
I20260807 11:30:16.554273 1113 leader_election.cc:288] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7182993a35ce4f5aa488a3caa7acea01 (127.24.103.126:42143), 53489d42e48a489eb63b33109c42b145 (127.24.103.124:36941)
I20260807 11:30:16.554440 1114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } }
I20260807 11:30:16.561723 1114 leader_election.cc:288] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7182993a35ce4f5aa488a3caa7acea01 (127.24.103.126:42143), 53f447304fdb45bf9a9f7ddf07bf91d3 (127.24.103.125:35933)
I20260807 11:30:16.570844 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.554472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:58464)
0807 11:30:16.570283 (+ 15811us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.570524 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15610,"threads_started":1}
I20260807 11:30:16.570883 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.555704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:58480)
0807 11:30:16.570283 (+ 14579us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.570519 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14396,"threads_started":1}
W20260807 11:30:16.571496 493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
W20260807 11:30:16.572248 32759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
I20260807 11:30:16.574277 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.573194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:58494)
0807 11:30:16.573808 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.573987 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":338,"threads_started":1}
W20260807 11:30:16.574577 493 leader_election.cc:334] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7182993a35ce4f5aa488a3caa7acea01 (127.24.103.126:42143): Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
I20260807 11:30:16.575904 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.575230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:58496)
0807 11:30:16.575457 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.575678 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 11:30:16.576154 32759 leader_election.cc:334] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7182993a35ce4f5aa488a3caa7acea01 (127.24.103.126:42143): Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
I20260807 11:30:16.578343 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.554586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:36941 (local address 127.0.0.1:59926)
0807 11:30:16.571463 (+ 16877us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.571482 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:16.571631 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:16.572112 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:16.572118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:16.572146 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:16.572475 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:16.572487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.574266 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.574273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:16.576398 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:16.576418 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.577786 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.577790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:16.577917 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.577941 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 11:30:16.578015 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 11:30:16.578149 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16756,"thread_start_us":16515,"threads_started":1}
I20260807 11:30:16.578699 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.554872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.24.103.124:36941)
0807 11:30:16.571234 (+ 16362us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:16.571243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:16.571770 (+ 527us) server_negotiation.cc:407] Connection header received
0807 11:30:16.571845 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:16.571853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:16.571930 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:16.572646 (+ 716us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:16.572860 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.574078 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.576560 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.577637 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.577786 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.577863 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:16.578079 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 11:30:16.578215 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16232,"thread_start_us":16374,"threads_started":1}
I20260807 11:30:16.579320 520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53489d42e48a489eb63b33109c42b145" is_pre_election: true
I20260807 11:30:16.579717 520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53f447304fdb45bf9a9f7ddf07bf91d3 in term 1.
I20260807 11:30:16.580317 32756 leader_election.cc:302] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53489d42e48a489eb63b33109c42b145, 53f447304fdb45bf9a9f7ddf07bf91d3; no voters: 7182993a35ce4f5aa488a3caa7acea01
I20260807 11:30:16.580763 1113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 11:30:16.580912 1113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7182993a35ce4f5aa488a3caa7acea01)
I20260807 11:30:16.581033 1113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:30:16.581458 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.571655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:35933 (local address 127.0.0.1:42746)
0807 11:30:16.571967 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.571986 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:16.572110 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:16.575592 (+ 3482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:16.575597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:16.575627 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:16.575781 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:16.575788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.577186 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.577197 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:16.579304 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:16.579319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.580711 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.580719 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:16.580902 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.580926 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 11:30:16.581027 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 11:30:16.581174 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260807 11:30:16.581593 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.573966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.24.103.125:35933)
0807 11:30:16.575238 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:16.575243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:16.575266 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:30:16.575318 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:16.575323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:16.575391 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:16.575892 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:16.576175 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.577025 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.579465 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.580531 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.580755 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.580832 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:16.581063 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 11:30:16.581198 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1140,"thread_start_us":927,"threads_started":1}
I20260807 11:30:16.582221 316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53489d42e48a489eb63b33109c42b145" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" is_pre_election: true
I20260807 11:30:16.583953 1113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } }
I20260807 11:30:16.584415 316 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 53489d42e48a489eb63b33109c42b145 in current term 2: Already voted for candidate 53f447304fdb45bf9a9f7ddf07bf91d3 in this term.
I20260807 11:30:16.584848 1113 leader_election.cc:288] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [CANDIDATE]: Term 2 election: Requested vote from peers 7182993a35ce4f5aa488a3caa7acea01 (127.24.103.126:42143), 53489d42e48a489eb63b33109c42b145 (127.24.103.124:36941)
I20260807 11:30:16.585420 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.584927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:58506)
0807 11:30:16.585216 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.585293 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260807 11:30:16.585417 520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53489d42e48a489eb63b33109c42b145"
I20260807 11:30:16.585479 490 leader_election.cc:302] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53489d42e48a489eb63b33109c42b145; no voters: 53f447304fdb45bf9a9f7ddf07bf91d3, 7182993a35ce4f5aa488a3caa7acea01
I20260807 11:30:16.585711 520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:30:16.586519 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.586150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:58518)
0807 11:30:16.586310 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.586401 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 11:30:16.586727 32759 leader_election.cc:334] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7182993a35ce4f5aa488a3caa7acea01 (127.24.103.126:42143): Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
I20260807 11:30:16.588258 520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f447304fdb45bf9a9f7ddf07bf91d3 in term 2.
I20260807 11:30:16.588459 1114 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 11:30:16.588706 32756 leader_election.cc:302] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53489d42e48a489eb63b33109c42b145, 53f447304fdb45bf9a9f7ddf07bf91d3; no voters: 7182993a35ce4f5aa488a3caa7acea01
I20260807 11:30:16.589022 1113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:30:16.589500 1113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [term 2 LEADER]: Becoming Leader. State: Replica: 53f447304fdb45bf9a9f7ddf07bf91d3, State: Running, Role: LEADER
I20260807 11:30:16.589923 1113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } }
I20260807 11:30:16.591943 1126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53f447304fdb45bf9a9f7ddf07bf91d3. Latest consensus state: current_term: 2 leader_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:16.592296 1126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:16.592935 1131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:30:16.597059 1131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:30:16.597978 1131 catalog_manager.cc:1259] Loaded cluster ID: 13e0d15ed169414d8d7b9cb933658be1
I20260807 11:30:16.598085 1131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:30:16.599390 1131 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:30:16.600289 1131 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: Loaded TSK: 0
I20260807 11:30:16.601220 1131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:30:16.624742 305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:30:16.631976 305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:30:16.636922 520 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [term 2 FOLLOWER]: Refusing update from remote peer 53f447304fdb45bf9a9f7ddf07bf91d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260807 11:30:16.637030 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.636586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:58532)
0807 11:30:16.636793 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.636875 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 11:30:16.638023 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.637669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:58534)
0807 11:30:16.637823 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.637891 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 11:30:16.637991 1126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260807 11:30:16.638301 32759 consensus_peers.cc:487] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 -> Peer 7182993a35ce4f5aa488a3caa7acea01 (127.24.103.126:42143): Couldn't send request to peer 7182993a35ce4f5aa488a3caa7acea01. Status: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 11:30:16.641664 1114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53f447304fdb45bf9a9f7ddf07bf91d3. Latest consensus state: current_term: 2 leader_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:16.641955 1114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:30:16.643991 1113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:16.644740 1113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:16.645454 1114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7182993a35ce4f5aa488a3caa7acea01" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42143 } } peers { permanent_uuid: "53f447304fdb45bf9a9f7ddf07bf91d3" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "53489d42e48a489eb63b33109c42b145" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 36941 } attrs { promote: false } } }
I20260807 11:30:16.645645 1114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53489d42e48a489eb63b33109c42b145 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:30:16.677193 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.652711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34573 (local address 127.0.0.1:45460)
0807 11:30:16.653179 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.653203 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:16.653349 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:16.667675 (+ 14326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:16.667685 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:16.667726 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:16.668168 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:16.668187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.671193 (+ 3006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.671198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:16.673635 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:16.673651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.676206 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.676212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:16.676412 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.676469 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 11:30:16.676597 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 11:30:16.676799 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":31}
I20260807 11:30:16.677500 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.652359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37329 (local address 127.0.0.1:34544)
0807 11:30:16.652578 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.652599 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:16.652712 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:16.668590 (+ 15878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:16.668600 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:16.668640 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:16.668907 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:16.668921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.670759 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.670765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:16.674868 (+ 4103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:16.674883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.676547 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.676556 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:16.676796 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.676828 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 11:30:16.676913 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 11:30:16.677069 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":2154}
I20260807 11:30:16.678086 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.652573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.24.103.67:37329)
0807 11:30:16.667981 (+ 15408us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:16.667991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:16.668019 (+ 28us) server_negotiation.cc:407] Connection header received
0807 11:30:16.668115 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:16.668124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:16.668361 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:16.669084 (+ 723us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:16.669444 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.670585 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.675095 (+ 4510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.676357 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.676494 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.676634 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:16.677285 (+ 651us) server_negotiation.cc:299] Negotiation successful
0807 11:30:16.677737 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15219,"thread_start_us":15645,"threads_started":1}
I20260807 11:30:16.678071 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.653151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.24.103.66:34573)
0807 11:30:16.667168 (+ 14017us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:16.667178 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:16.667211 (+ 33us) server_negotiation.cc:407] Connection header received
0807 11:30:16.667316 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:16.667324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:16.667434 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:16.668374 (+ 940us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:16.668598 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.670045 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.675147 (+ 5102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.675952 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.676127 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.676251 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:16.677586 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0807 11:30:16.677764 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13800,"thread_start_us":13504,"threads_started":1}
I20260807 11:30:16.679104 803 tablet_service.cc:1467] Processing CreateTablet for tablet ef19cec1ceba417c875e2f7372fb24d2 (DEFAULT_TABLE table=test-table [id=eb880cc471074a60bc33481d5cd77ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:16.679600 803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef19cec1ceba417c875e2f7372fb24d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:16.679746 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.653892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.24.103.65:46611)
0807 11:30:16.670053 (+ 16161us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:16.670059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:16.671683 (+ 1624us) server_negotiation.cc:407] Connection header received
0807 11:30:16.671774 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:16.671779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:16.671864 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:16.671970 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:16.673259 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.674423 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.676978 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.678517 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.678678 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.678790 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:16.679283 (+ 493us) server_negotiation.cc:299] Negotiation successful
0807 11:30:16.679427 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16015,"thread_start_us":16143,"threads_started":1}
I20260807 11:30:16.679766 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.653173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:46611 (local address 127.0.0.1:36396)
0807 11:30:16.670584 (+ 17411us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.670611 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:16.670738 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:16.672291 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:16.672297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:16.672333 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:16.672813 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:16.672824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.674580 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.674584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:16.676818 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:16.676830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.678745 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.678883 (+ 138us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:16.679083 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.679130 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 11:30:16.679237 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 11:30:16.679365 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17285,"mutex_wait_us":5,"thread_start_us":17091,"threads_started":1}
I20260807 11:30:16.680621 636 tablet_service.cc:1467] Processing CreateTablet for tablet ef19cec1ceba417c875e2f7372fb24d2 (DEFAULT_TABLE table=test-table [id=eb880cc471074a60bc33481d5cd77ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:16.681077 636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef19cec1ceba417c875e2f7372fb24d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:16.683787 957 tablet_service.cc:1467] Processing CreateTablet for tablet ef19cec1ceba417c875e2f7372fb24d2 (DEFAULT_TABLE table=test-table [id=eb880cc471074a60bc33481d5cd77ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:16.684211 957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef19cec1ceba417c875e2f7372fb24d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:16.687032 1141 tablet_bootstrap.cc:492] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8: Bootstrap starting.
I20260807 11:30:16.688472 1142 tablet_bootstrap.cc:492] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05: Bootstrap starting.
I20260807 11:30:16.689214 1141 tablet_bootstrap.cc:654] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:16.690014 1142 tablet_bootstrap.cc:654] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:16.690797 1144 tablet_bootstrap.cc:492] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6: Bootstrap starting.
I20260807 11:30:16.692579 1142 tablet_bootstrap.cc:492] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05: No bootstrap required, opened a new log
I20260807 11:30:16.692603 1144 tablet_bootstrap.cc:654] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:16.692741 1142 ts_tablet_manager.cc:1397] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260807 11:30:16.693624 1142 raft_consensus.cc:348] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } } peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } }
I20260807 11:30:16.693912 1142 raft_consensus.cc:374] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:16.694067 1142 raft_consensus.cc:729] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7422a6ab063542509f75e3098e0dca05, State: Initialized, Role: FOLLOWER
I20260807 11:30:16.694389 1142 consensus_queue.cc:260] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } } peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } }
I20260807 11:30:16.695073 1144 tablet_bootstrap.cc:492] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6: No bootstrap required, opened a new log
I20260807 11:30:16.695305 1144 ts_tablet_manager.cc:1397] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260807 11:30:16.696436 1144 raft_consensus.cc:348] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } } peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } }
I20260807 11:30:16.696754 1144 raft_consensus.cc:374] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:16.696874 1144 raft_consensus.cc:729] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae8abbb0c9242f49c9dd17ad13de1a6, State: Initialized, Role: FOLLOWER
I20260807 11:30:16.697162 1144 consensus_queue.cc:260] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } } peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } }
I20260807 11:30:16.698096 1144 ts_tablet_manager.cc:1428] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 11:30:16.699568 1141 tablet_bootstrap.cc:492] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8: No bootstrap required, opened a new log
I20260807 11:30:16.699805 1141 ts_tablet_manager.cc:1397] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260807 11:30:16.700724 1141 raft_consensus.cc:348] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } } peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } }
I20260807 11:30:16.701081 1141 raft_consensus.cc:374] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:16.701241 1141 raft_consensus.cc:729] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c004b483df4454b90ba9594e4625e8, State: Initialized, Role: FOLLOWER
I20260807 11:30:16.701575 1141 consensus_queue.cc:260] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } } peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } }
I20260807 11:30:16.703692 1142 ts_tablet_manager.cc:1428] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05: Time spent starting tablet: real 0.011s user 0.000s sys 0.008s
I20260807 11:30:16.705474 1141 ts_tablet_manager.cc:1428] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260807 11:30:16.706633 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.704251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.65:53155)
0807 11:30:16.706157 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.706348 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1755,"threads_started":1}
I20260807 11:30:16.707679 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.698480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.67:55093)
0807 11:30:16.707282 (+ 8802us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.707419 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8665,"thread_start_us":8525,"threads_started":1}
I20260807 11:30:16.708292 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.707216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.66:50933)
0807 11:30:16.707859 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.708037 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"threads_started":1}
I20260807 11:30:16.709749 713 heartbeater.cc:503] Master 127.24.103.125:35933 was elected leader, sending a full tablet report...
I20260807 11:30:16.709851 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.709437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.67:36699)
0807 11:30:16.709604 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.709710 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 11:30:16.710542 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.709971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.65:53363)
0807 11:30:16.710291 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.710385 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260807 11:30:16.753325 1149 raft_consensus.cc:482] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:30:16.753623 1149 raft_consensus.cc:504] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } } peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } }
I20260807 11:30:16.754406 1149 leader_election.cc:288] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62c004b483df4454b90ba9594e4625e8 (127.24.103.66:34573), 7422a6ab063542509f75e3098e0dca05 (127.24.103.65:46611)
I20260807 11:30:16.755183 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef19cec1ceba417c875e2f7372fb24d2" candidate_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7422a6ab063542509f75e3098e0dca05" is_pre_election: true
I20260807 11:30:16.755203 823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef19cec1ceba417c875e2f7372fb24d2" candidate_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c004b483df4454b90ba9594e4625e8" is_pre_election: true
I20260807 11:30:16.755486 823 raft_consensus.cc:2454] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eae8abbb0c9242f49c9dd17ad13de1a6 in term 0.
I20260807 11:30:16.755487 659 raft_consensus.cc:2454] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eae8abbb0c9242f49c9dd17ad13de1a6 in term 0.
I20260807 11:30:16.756054 904 leader_election.cc:302] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c004b483df4454b90ba9594e4625e8, eae8abbb0c9242f49c9dd17ad13de1a6; no voters:
I20260807 11:30:16.756345 1149 raft_consensus.cc:2781] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:30:16.756487 1149 raft_consensus.cc:482] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:30:16.756611 1149 raft_consensus.cc:3037] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:16.758842 1149 raft_consensus.cc:504] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } } peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } }
I20260807 11:30:16.759467 1149 leader_election.cc:288] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [CANDIDATE]: Term 1 election: Requested vote from peers 62c004b483df4454b90ba9594e4625e8 (127.24.103.66:34573), 7422a6ab063542509f75e3098e0dca05 (127.24.103.65:46611)
I20260807 11:30:16.759975 823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef19cec1ceba417c875e2f7372fb24d2" candidate_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c004b483df4454b90ba9594e4625e8"
I20260807 11:30:16.760007 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef19cec1ceba417c875e2f7372fb24d2" candidate_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7422a6ab063542509f75e3098e0dca05"
I20260807 11:30:16.760192 823 raft_consensus.cc:3037] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:16.760232 659 raft_consensus.cc:3037] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:16.762282 659 raft_consensus.cc:2454] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eae8abbb0c9242f49c9dd17ad13de1a6 in term 1.
I20260807 11:30:16.762310 823 raft_consensus.cc:2454] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eae8abbb0c9242f49c9dd17ad13de1a6 in term 1.
I20260807 11:30:16.762722 905 leader_election.cc:302] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7422a6ab063542509f75e3098e0dca05, eae8abbb0c9242f49c9dd17ad13de1a6; no voters:
I20260807 11:30:16.763058 1149 raft_consensus.cc:2781] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:16.763531 1149 raft_consensus.cc:686] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 1 LEADER]: Becoming Leader. State: Replica: eae8abbb0c9242f49c9dd17ad13de1a6, State: Running, Role: LEADER
I20260807 11:30:16.763859 1149 consensus_queue.cc:237] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } } peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } }
I20260807 11:30:16.764050 1031 heartbeater.cc:503] Master 127.24.103.125:35933 was elected leader, sending a full tablet report...
I20260807 11:30:16.764662 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.764202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.67:43503)
0807 11:30:16.764387 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.764501 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 11:30:16.767050 304 catalog_manager.cc:5495] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eae8abbb0c9242f49c9dd17ad13de1a6 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae8abbb0c9242f49c9dd17ad13de1a6" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 } health_report { overall_health: UNKNOWN } } }
I20260807 11:30:16.792997 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.788427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.24.103.67:37329)
0807 11:30:16.788616 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:16.788620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:16.788639 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:30:16.788749 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:16.788752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:16.788817 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:16.788895 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:30:16.789352 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.790068 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.791416 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.791843 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.791940 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.792373 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:30:16.792480 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:30:16.792563 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:16.792730 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 11:30:16.792797 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 11:30:16.796334 823 raft_consensus.cc:1261] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8 [term 1 FOLLOWER]: Refusing update from remote peer eae8abbb0c9242f49c9dd17ad13de1a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:16.796362 659 raft_consensus.cc:1261] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05 [term 1 FOLLOWER]: Refusing update from remote peer eae8abbb0c9242f49c9dd17ad13de1a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:16.797060 1154 consensus_queue.cc:1035] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c004b483df4454b90ba9594e4625e8" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:16.797462 1149 consensus_queue.cc:1035] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7422a6ab063542509f75e3098e0dca05" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:16.799702 874 heartbeater.cc:503] Master 127.24.103.125:35933 was elected leader, sending a full tablet report...
I20260807 11:30:16.800001 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.799489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.65:42703)
0807 11:30:16.799671 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.799786 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 11:30:16.800407 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.799931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.66:45405)
0807 11:30:16.800173 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.800277 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260807 11:30:16.807555 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.806901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.65:55273)
0807 11:30:16.807200 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.807320 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260807 11:30:16.815053 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.814432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.67:42183)
0807 11:30:16.814728 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.814827 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260807 11:30:16.822579 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.822044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.66:59739)
0807 11:30:16.822271 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.822364 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260807 11:30:16.823885 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.823150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.24.103.67:55357)
0807 11:30:16.823498 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:16.823734 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
I20260807 11:30:16.824666 305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:16.824910 305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:16.830338 305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53f447304fdb45bf9a9f7ddf07bf91d3: Sending DeleteTablet for 3 replicas of tablet ef19cec1ceba417c875e2f7372fb24d2
I20260807 11:30:16.831525 957 tablet_service.cc:1514] Processing DeleteTablet for tablet ef19cec1ceba417c875e2f7372fb24d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:16 UTC) from {username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:34544
I20260807 11:30:16.831507 803 tablet_service.cc:1514] Processing DeleteTablet for tablet ef19cec1ceba417c875e2f7372fb24d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:16 UTC) from {username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:45460
I20260807 11:30:16.831616 636 tablet_service.cc:1514] Processing DeleteTablet for tablet ef19cec1ceba417c875e2f7372fb24d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:16 UTC) from {username='oryx', principal='oryx/127.24.103.125@KRBTEST.COM'} at 127.0.0.1:36396
I20260807 11:30:16.832662 1168 tablet_replica.cc:330] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6: stopping tablet replica
I20260807 11:30:16.832897 1168 raft_consensus.cc:2229] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:30:16.834096 1168 raft_consensus.cc:2258] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:16.834542 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 501
I20260807 11:30:16.835729 1168 ts_tablet_manager.cc:1905] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:16.845081 1168 ts_tablet_manager.cc:1918] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:16.845683 1169 tablet_replica.cc:330] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05: stopping tablet replica
I20260807 11:30:16.845798 1167 tablet_replica.cc:330] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8: stopping tablet replica
I20260807 11:30:16.845916 1169 raft_consensus.cc:2229] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:16.846035 1167 raft_consensus.cc:2229] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:16.846325 1169 raft_consensus.cc:2258] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:16.846376 1167 raft_consensus.cc:2258] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:16.847486 1169 ts_tablet_manager.cc:1905] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:16.847653 1167 ts_tablet_manager.cc:1905] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:16.853523 1167 ts_tablet_manager.cc:1918] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:16.853605 1169 ts_tablet_manager.cc:1918] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:16.853684 1167 log.cc:1196] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/ef19cec1ceba417c875e2f7372fb24d2
I20260807 11:30:16.853834 1169 log.cc:1196] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/ef19cec1ceba417c875e2f7372fb24d2
I20260807 11:30:16.854261 1169 ts_tablet_manager.cc:1939] T ef19cec1ceba417c875e2f7372fb24d2 P 7422a6ab063542509f75e3098e0dca05: Deleting consensus metadata
I20260807 11:30:16.854277 1167 ts_tablet_manager.cc:1939] T ef19cec1ceba417c875e2f7372fb24d2 P 62c004b483df4454b90ba9594e4625e8: Deleting consensus metadata
I20260807 11:30:16.855186 1168 log.cc:1196] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/ef19cec1ceba417c875e2f7372fb24d2
I20260807 11:30:16.855611 32757 catalog_manager.cc:4841] TS 7422a6ab063542509f75e3098e0dca05 (127.24.103.65:46611): tablet ef19cec1ceba417c875e2f7372fb24d2 (table test-table [id=eb880cc471074a60bc33481d5cd77ecc]) successfully deleted
I20260807 11:30:16.855825 1168 ts_tablet_manager.cc:1939] T ef19cec1ceba417c875e2f7372fb24d2 P eae8abbb0c9242f49c9dd17ad13de1a6: Deleting consensus metadata
I20260807 11:30:16.856243 32756 catalog_manager.cc:4841] TS 62c004b483df4454b90ba9594e4625e8 (127.24.103.66:34573): tablet ef19cec1ceba417c875e2f7372fb24d2 (table test-table [id=eb880cc471074a60bc33481d5cd77ecc]) successfully deleted
I20260807 11:30:16.857056 32756 catalog_manager.cc:4841] TS eae8abbb0c9242f49c9dd17ad13de1a6 (127.24.103.67:37329): tablet ef19cec1ceba417c875e2f7372fb24d2 (table test-table [id=eb880cc471074a60bc33481d5cd77ecc]) successfully deleted
I20260807 11:30:16.861835 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.836477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.24.103.65:46611)
0807 11:30:16.836744 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:16.836754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:16.845278 (+ 8524us) server_negotiation.cc:407] Connection header received
0807 11:30:16.845739 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:16.845747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:16.845861 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:16.845986 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:16.847999 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.849105 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.850898 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.851636 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.851791 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.855502 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:16.855526 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:16.855549 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:16.855598 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:16.858843 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:16.859718 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:16.859726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:16.859728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:16.859870 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:16.860322 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:16.860325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:16.860327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:16.860684 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:16.860935 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:16.861217 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 11:30:16.861397 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:30:16.862504 704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:16.958243 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 501
I20260807 11:30:16.981348 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 723
I20260807 11:30:16.995640 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:16.982209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.24.103.66:34573)
0807 11:30:16.982434 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:16.982441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:16.982519 (+ 78us) server_negotiation.cc:407] Connection header received
0807 11:30:16.982774 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:16.982779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:16.982845 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:16.982954 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:16.984371 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.985068 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.986315 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:16.986743 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:16.986878 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:16.989550 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:16.989565 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:16.989583 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:16.989620 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:16.992668 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:16.993431 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:16.993444 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:16.993446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:16.993579 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:16.993961 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:16.993967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:16.993971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:16.994365 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:16.994723 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:16.995036 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 11:30:16.995290 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:30:16.996495 866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:17.091820 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 723
I20260807 11:30:17.111758 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 886
I20260807 11:30:17.126614 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:17.112619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.24.103.67:37329)
0807 11:30:17.112828 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:17.112833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:17.112857 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:30:17.112986 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:17.112990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:17.113072 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:17.113175 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:17.114606 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:17.115493 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:17.116899 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:17.117315 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:17.117415 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:17.120487 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:17.120511 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:17.120527 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:17.120562 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:17.123833 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:17.124656 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:17.124663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:17.124666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:17.124810 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:17.125222 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:17.125235 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:17.125237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:17.125612 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:17.125861 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:17.126155 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 11:30:17.126329 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:30:17.127468 1024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:17.185243 1174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:17.184449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:58536)
0807 11:30:17.184906 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:17.185035 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":222,"threads_started":1}
I20260807 11:30:17.188259 1174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:17.187801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42143 (local address 127.0.0.1:58542)
0807 11:30:17.187976 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:17.188081 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 11:30:17.253326 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 886
I20260807 11:30:17.281183 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 32619
I20260807 11:30:17.282502 337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:17.419402 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 32619
W20260807 11:30:17.445948 490 connection.cc:535] server connection from 127.0.0.1:59926 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 11:30:17.446329 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 32728
I20260807 11:30:17.447472 541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:17.568550 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 32728
2026-08-07T11:30:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10102 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:30:17 dist-test-slave-wl2h krb5kdc[1180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:30:17 dist-test-slave-wl2h krb5kdc[1180](info): set up 2 sockets
Aug 07 11:30:17 dist-test-slave-wl2h krb5kdc[1180](info): commencing operation
krb5kdc: starting...
W20260807 11:30:18.794504 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:30:18 dist-test-slave-wl2h krb5kdc[1180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:30:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:30:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.126.keytab.
Entry for principal oryx/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.126.keytab.
I20260807 11:30:19.084159 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44615
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.126.keytab
--principal=oryx/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:19.449213 1193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:19.449509 1193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:19.449681 1193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:19.457082 1193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:19.457203 1193 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 11:30:19.457290 1193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:19.457412 1193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:19.457484 1193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:19.457573 1193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:30:19.467949 1193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:44615
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.126.keytab
--principal=oryx/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:19.469386 1193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:19.470867 1193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:19.484077 1202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:19.484082 1199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:19.484056 1200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:19.485025 1193 server_base.cc:1034] running on GCE node
I20260807 11:30:19.485900 1193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:19.487429 1193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:19.488704 1193 hybrid_clock.cc:648] HybridClock initialized: now 1786102219488605 us; error 64 us; skew 500 ppm
I20260807 11:30:19.489046 1193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:19 dist-test-slave-wl2h krb5kdc[1180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102219, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:19.492396 1193 init.cc:377] Logged in from keytab as oryx/127.24.103.126@KRBTEST.COM (short username oryx)
I20260807 11:30:19.493891 1193 webserver.cc:460] Webserver started at http://127.24.103.126:46845/ using document root <none> and password file <none>
I20260807 11:30:19.494613 1193 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:19.494714 1193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:19.495131 1193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:19.498140 1193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "a3044f1c76144a75a313adab9d4c3949"
format_stamp: "Formatted at 2026-08-07 11:30:19 on dist-test-slave-wl2h"
server_key: "d2e63ea446cf604c229a0117e71ff9e0"
server_key_iv: "81d78fcc2ab11b8aa4174e8a72839a97"
server_key_version: "encryptionkey@0"
I20260807 11:30:19.498986 1193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "a3044f1c76144a75a313adab9d4c3949"
format_stamp: "Formatted at 2026-08-07 11:30:19 on dist-test-slave-wl2h"
server_key: "d2e63ea446cf604c229a0117e71ff9e0"
server_key_iv: "81d78fcc2ab11b8aa4174e8a72839a97"
server_key_version: "encryptionkey@0"
I20260807 11:30:19.504281 1193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 11:30:19.507959 1210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:19.509320 1193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:30:19.509625 1193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "a3044f1c76144a75a313adab9d4c3949"
format_stamp: "Formatted at 2026-08-07 11:30:19 on dist-test-slave-wl2h"
server_key: "d2e63ea446cf604c229a0117e71ff9e0"
server_key_iv: "81d78fcc2ab11b8aa4174e8a72839a97"
server_key_version: "encryptionkey@0"
I20260807 11:30:19.509865 1193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:19.528246 1193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:19.533006 1193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:19.533303 1193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:19.555897 1193 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:44615
I20260807 11:30:19.555904 1279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:44615 every 8 connection(s)
I20260807 11:30:19.557320 1193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:30:19.561250 1280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:19.563848 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 1193
I20260807 11:30:19.564085 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:30:19.564744 24989 external_mini_cluster.cc:1428] Setting key f8cc148e6ce54a6608b02b3dcd35d3ca
I20260807 11:30:19.574119 1280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949: Bootstrap starting.
I20260807 11:30:19.578310 1280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:19.579797 1280 log.cc:824] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949: Log is configured to *not* fsync() on all Append() calls
Aug 07 11:30:19 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:19.583206 1280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949: No bootstrap required, opened a new log
I20260807 11:30:19.590334 1284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:19.566833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.24.103.126:44615)
0807 11:30:19.567679 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:19.567691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:19.567733 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:30:19.569565 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:19.569588 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:19.569986 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:19.570323 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:19.572397 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:19.573820 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:19.575597 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:19.576361 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:19.576507 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:19.581907 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:19.581915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:19.581929 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:19.581967 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:19.585402 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:19.586628 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:19.586635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:19.586642 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:19.586842 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:19.587372 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:19.587376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:19.587379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:19.587985 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:19.588289 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:19.588737 (+ 448us) server_negotiation.cc:299] Negotiation successful
0807 11:30:19.589161 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":410,"threads_started":1}
I20260807 11:30:19.593116 1280 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3044f1c76144a75a313adab9d4c3949" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44615 } }
I20260807 11:30:19.593603 1280 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:19.593713 1280 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3044f1c76144a75a313adab9d4c3949, State: Initialized, Role: FOLLOWER
I20260807 11:30:19.594353 1280 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3044f1c76144a75a313adab9d4c3949" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44615 } }
I20260807 11:30:19.594591 1280 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:30:19.594730 1280 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:30:19.594866 1280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:19.597760 1280 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3044f1c76144a75a313adab9d4c3949" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44615 } }
I20260807 11:30:19.598380 1280 leader_election.cc:302] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3044f1c76144a75a313adab9d4c3949; no voters:
I20260807 11:30:19.599081 1280 leader_election.cc:288] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:30:19.599244 1286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:19.599997 1286 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [term 1 LEADER]: Becoming Leader. State: Replica: a3044f1c76144a75a313adab9d4c3949, State: Running, Role: LEADER
I20260807 11:30:19.600562 1286 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3044f1c76144a75a313adab9d4c3949" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44615 } }
I20260807 11:30:19.601297 1280 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:19.603477 1288 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3044f1c76144a75a313adab9d4c3949. Latest consensus state: current_term: 1 leader_uuid: "a3044f1c76144a75a313adab9d4c3949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3044f1c76144a75a313adab9d4c3949" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44615 } } }
I20260807 11:30:19.603891 1288 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:19.607304 1287 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3044f1c76144a75a313adab9d4c3949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3044f1c76144a75a313adab9d4c3949" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44615 } } }
I20260807 11:30:19.607686 1287 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:19.608417 1295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:30:19.612984 1295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:30:19.622658 1295 catalog_manager.cc:1347] Generated new cluster ID: 91d15cd5bd934c23aada08d3db486d06
I20260807 11:30:19.622787 1295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:30:19.646809 1295 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:30:19.648901 1295 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:30:19.664345 1295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949: Generated new TSK 0
I20260807 11:30:19.665288 1295 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
I20260807 11:30:19.740439 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44615
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:20.101259 1309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:20.101601 1309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:20.101773 1309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:20.110029 1309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:20.110211 1309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:20.110360 1309 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:30:20.110502 1309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:30:20.121073 1309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:20.122503 1309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:20.123958 1309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:20.138221 1314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:20.138149 1315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:20.138293 1317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:20.139559 1309 server_base.cc:1034] running on GCE node
I20260807 11:30:20.140319 1309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:20.141428 1309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:20.142705 1309 hybrid_clock.cc:648] HybridClock initialized: now 1786102220142636 us; error 42 us; skew 500 ppm
I20260807 11:30:20.143160 1309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:20 dist-test-slave-wl2h krb5kdc[1180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102220, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:20.152875 1309 init.cc:377] Logged in from keytab as oryx/127.24.103.65@KRBTEST.COM (short username oryx)
I20260807 11:30:20.154597 1309 webserver.cc:460] Webserver started at http://127.24.103.65:45803/ using document root <none> and password file <none>
I20260807 11:30:20.155414 1309 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:20.155558 1309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:20.155896 1309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:20.158915 1309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc"
format_stamp: "Formatted at 2026-08-07 11:30:20 on dist-test-slave-wl2h"
server_key: "dd2d1fb8799120e3f6d990feae8484e6"
server_key_iv: "02689061993c200b34319487ff0f2487"
server_key_version: "encryptionkey@0"
I20260807 11:30:20.159816 1309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc"
format_stamp: "Formatted at 2026-08-07 11:30:20 on dist-test-slave-wl2h"
server_key: "dd2d1fb8799120e3f6d990feae8484e6"
server_key_iv: "02689061993c200b34319487ff0f2487"
server_key_version: "encryptionkey@0"
I20260807 11:30:20.165287 1309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:30:20.169109 1324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:20.170786 1309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:30:20.170977 1309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc"
format_stamp: "Formatted at 2026-08-07 11:30:20 on dist-test-slave-wl2h"
server_key: "dd2d1fb8799120e3f6d990feae8484e6"
server_key_iv: "02689061993c200b34319487ff0f2487"
server_key_version: "encryptionkey@0"
I20260807 11:30:20.171293 1309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:20.189378 1309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:20.194124 1309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:20.194427 1309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:20.197345 1309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:20.197468 1309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:20.197665 1309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:20.197732 1309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:30:20 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102220, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:20.226014 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:20.198252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:44699 (local address 127.24.103.126:44615)
0807 11:30:20.198796 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:20.198803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:20.200266 (+ 1463us) server_negotiation.cc:407] Connection header received
0807 11:30:20.202901 (+ 2635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:20.202912 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:20.203079 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:20.203225 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:20.205867 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.207230 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.210169 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.210861 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.211056 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:20.217501 (+ 6445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:20.217511 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:20.217519 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:20.217583 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:20.221044 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:20.222337 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:20.222341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:20.222344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:20.222485 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:20.223239 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:20.223247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:20.223252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:20.223654 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:20.223870 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:20.225076 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0807 11:30:20.225396 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":256,"threads_started":1}
I20260807 11:30:20.227985 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:20.198990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44615 (local address 127.24.103.65:44699)
0807 11:30:20.200063 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:20.200118 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:20.202420 (+ 2302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:20.203551 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:20.203560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:20.204550 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:20.205612 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:20.205651 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.207463 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.207469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:20.209904 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:20.209929 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.211207 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.211217 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:20.211635 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:20.212609 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:20.212659 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:20.217142 (+ 4483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:20.221264 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:20.221279 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:20.221307 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:20.222169 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:20.222646 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:20.222654 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:20.222660 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:20.223069 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:20.223871 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:20.223881 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:20.224535 (+ 654us) client_negotiation.cc:769] Sending connection context
0807 11:30:20.224908 (+ 373us) client_negotiation.cc:240] Negotiation successful
0807 11:30:20.225860 (+ 952us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":427,"threads_started":1}
I20260807 11:30:20.247958 1309 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:35005
I20260807 11:30:20.248003 1459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:35005 every 8 connection(s)
I20260807 11:30:20.249285 1309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:30:20.251576 1460 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44615
I20260807 11:30:20.251828 1460 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:20.251902 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 1309
I20260807 11:30:20.252146 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:30:20.252599 1460 heartbeater.cc:511] Master 127.24.103.126:44615 requested a full tablet report, sending...
I20260807 11:30:20.252862 24989 external_mini_cluster.cc:1428] Setting key f707359253bb0ac9dcf3bad484aeaecc
I20260807 11:30:20.254961 1228 ts_manager.cc:194] Registered new tserver with Master: bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65:35005)
I20260807 11:30:20.255223 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:20.238212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:43251 (local address 127.24.103.126:44615)
0807 11:30:20.238399 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:20.238404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:20.239007 (+ 603us) server_negotiation.cc:407] Connection header received
0807 11:30:20.239268 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:20.239273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:20.239366 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:20.239475 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:20.240919 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.241837 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.244064 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.244721 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.244892 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:20.247322 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:20.247327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:20.247330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:20.247363 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:20.250156 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:20.251078 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:20.251084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:20.251088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:20.251237 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:20.253501 (+ 2264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:20.253506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:20.253510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:20.253925 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:20.254120 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:20.254528 (+ 408us) server_negotiation.cc:299] Negotiation successful
0807 11:30:20.254769 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 11:30:20.255478 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:20.238243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44615 (local address 127.24.103.65:43251)
0807 11:30:20.238888 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:20.238905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:20.239128 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:20.239516 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:20.239520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:20.240171 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:20.240698 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:20.240717 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.242038 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.242042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:20.243911 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:20.243927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.245171 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.245181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:20.245305 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:20.246116 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:20.246138 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:20.247126 (+ 988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:20.250338 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:20.250345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:20.250353 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:20.250900 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:20.251354 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:20.251359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:20.251362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:20.251535 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:20.254052 (+ 2517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:20.254060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:20.254252 (+ 192us) client_negotiation.cc:769] Sending connection context
0807 11:30:20.254395 (+ 143us) client_negotiation.cc:240] Negotiation successful
0807 11:30:20.254736 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":370,"threads_started":1}
I20260807 11:30:20.257185 1228 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:44699
WARNING: no policy specified for oryx/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
I20260807 11:30:20.318737 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44615
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:20.675979 1464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:20.676285 1464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:20.676453 1464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:20.684223 1464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:20.684392 1464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:20.684541 1464 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:30:20.684684 1464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:30:20.696247 1464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:20.697666 1464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:20.699115 1464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:20.715230 1471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:20.715282 1473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:20.715306 1469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:20.716116 1464 server_base.cc:1034] running on GCE node
I20260807 11:30:20.716820 1464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:20.717794 1464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:20.719163 1464 hybrid_clock.cc:648] HybridClock initialized: now 1786102220719060 us; error 54 us; skew 500 ppm
I20260807 11:30:20.719517 1464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:20 dist-test-slave-wl2h krb5kdc[1180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102220, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:20.723006 1464 init.cc:377] Logged in from keytab as oryx/127.24.103.66@KRBTEST.COM (short username oryx)
I20260807 11:30:20.724709 1464 webserver.cc:460] Webserver started at http://127.24.103.66:42463/ using document root <none> and password file <none>
I20260807 11:30:20.725445 1464 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:20.725536 1464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:20.725805 1464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:20.728839 1464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "92f47045eb9d47a2a8da13f1f5598e18"
format_stamp: "Formatted at 2026-08-07 11:30:20 on dist-test-slave-wl2h"
server_key: "6d8c7be16954a2bafc2361b1b36c06e5"
server_key_iv: "6fcdfec7d0b38bb1133e6d8310111731"
server_key_version: "encryptionkey@0"
I20260807 11:30:20.729698 1464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "92f47045eb9d47a2a8da13f1f5598e18"
format_stamp: "Formatted at 2026-08-07 11:30:20 on dist-test-slave-wl2h"
server_key: "6d8c7be16954a2bafc2361b1b36c06e5"
server_key_iv: "6fcdfec7d0b38bb1133e6d8310111731"
server_key_version: "encryptionkey@0"
I20260807 11:30:20.735452 1464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 11:30:20.739697 1480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:20.741262 1464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:30:20.741478 1464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "92f47045eb9d47a2a8da13f1f5598e18"
format_stamp: "Formatted at 2026-08-07 11:30:20 on dist-test-slave-wl2h"
server_key: "6d8c7be16954a2bafc2361b1b36c06e5"
server_key_iv: "6fcdfec7d0b38bb1133e6d8310111731"
server_key_version: "encryptionkey@0"
I20260807 11:30:20.741776 1464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:20.770802 1464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:20.776979 1464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:20.777350 1464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:20.780818 1464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:20.780956 1464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:20.781134 1464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:20.781219 1464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:30:20 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102220, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:20.808904 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:20.781770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:33589 (local address 127.24.103.126:44615)
0807 11:30:20.782286 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:20.782293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:20.783826 (+ 1533us) server_negotiation.cc:407] Connection header received
0807 11:30:20.786672 (+ 2846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:20.786679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:20.786808 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:20.786953 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:20.789177 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.790184 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.792855 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.793560 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.793748 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:20.800032 (+ 6284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:20.800043 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:20.800048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:20.800108 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:20.804185 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:20.805491 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:20.805500 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:20.805506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:20.805828 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:20.806594 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:20.806600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:20.806604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:20.807102 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:20.807305 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:20.808186 (+ 881us) server_negotiation.cc:299] Negotiation successful
0807 11:30:20.808451 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":244,"threads_started":1}
I20260807 11:30:20.810074 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:20.782505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44615 (local address 127.24.103.66:33589)
0807 11:30:20.783582 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:20.783647 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:20.786215 (+ 2568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:20.787277 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:20.787286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:20.788325 (+ 1039us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:20.788970 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:20.788989 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.790378 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.790383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:20.792639 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:20.792663 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.794103 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.794109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:20.794570 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:20.795558 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:20.795584 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:20.799596 (+ 4012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:20.804375 (+ 4779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:20.804387 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:20.804418 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:20.805285 (+ 867us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:20.806023 (+ 738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:20.806030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:20.806036 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:20.806443 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:20.807269 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:20.807279 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:20.807705 (+ 426us) client_negotiation.cc:769] Sending connection context
0807 11:30:20.808076 (+ 371us) client_negotiation.cc:240] Negotiation successful
0807 11:30:20.808690 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":488,"threads_started":1}
I20260807 11:30:20.837178 1464 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:40989
I20260807 11:30:20.837292 1614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:40989 every 8 connection(s)
I20260807 11:30:20.838747 1464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:30:20.840905 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 1464
I20260807 11:30:20.841133 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:30:20.841797 24989 external_mini_cluster.cc:1428] Setting key 47a651cb437e8890d6094b9b99462ccf
I20260807 11:30:20.841761 1615 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44615
I20260807 11:30:20.842123 1615 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:20.842173 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:20.821650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:40515 (local address 127.24.103.126:44615)
0807 11:30:20.821880 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:20.821888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:20.822949 (+ 1061us) server_negotiation.cc:407] Connection header received
0807 11:30:20.823255 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:20.823266 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:20.823378 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:20.823509 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:20.825706 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.826965 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.829639 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.830539 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.830708 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:20.834291 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:20.834301 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:20.834308 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:20.834362 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:20.838283 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:20.839261 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:20.839265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:20.839268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:20.839477 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:20.840096 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:20.840101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:20.840103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:20.840668 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:20.840867 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:20.841500 (+ 633us) server_negotiation.cc:299] Negotiation successful
0807 11:30:20.841774 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:30:20.842403 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:20.822061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44615 (local address 127.24.103.66:40515)
0807 11:30:20.822787 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:20.822815 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:20.823042 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:20.823775 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:20.823782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:20.824792 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:20.825441 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:20.825464 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.827164 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.827170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:20.829415 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:20.829442 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:20.830926 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:20.830940 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:20.831279 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:20.832198 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:20.832230 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:20.834015 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:20.838441 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:20.838447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:20.838451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:20.839129 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:20.839628 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:20.839636 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:20.839642 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:20.839867 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:20.840830 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:20.840835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:20.841130 (+ 295us) client_negotiation.cc:769] Sending connection context
0807 11:30:20.841354 (+ 224us) client_negotiation.cc:240] Negotiation successful
0807 11:30:20.841785 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":415,"threads_started":1}
I20260807 11:30:20.843926 1615 heartbeater.cc:511] Master 127.24.103.126:44615 requested a full tablet report, sending...
I20260807 11:30:20.845773 1227 ts_manager.cc:194] Registered new tserver with Master: 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989)
I20260807 11:30:20.846961 1227 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:33589
WARNING: no policy specified for oryx/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
I20260807 11:30:20.907815 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44615
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:21.258788 1619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:21.259145 1619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:21.259264 1619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 11:30:21.261186 1460 heartbeater.cc:503] Master 127.24.103.126:44615 was elected leader, sending a full tablet report...
W20260807 11:30:21.267278 1619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:21.267383 1619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:21.267468 1619 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 11:30:21.267552 1619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:30:21.277987 1619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.103.126:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:21.279418 1619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:21.280767 1619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:21.294713 1624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:21.294773 1627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:21.294790 1625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:21.295740 1619 server_base.cc:1034] running on GCE node
I20260807 11:30:21.296396 1619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:21.297399 1619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:21.298707 1619 hybrid_clock.cc:648] HybridClock initialized: now 1786102221298623 us; error 42 us; skew 500 ppm
I20260807 11:30:21.299120 1619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:21 dist-test-slave-wl2h krb5kdc[1180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102221, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:21.302563 1619 init.cc:377] Logged in from keytab as oryx/127.24.103.67@KRBTEST.COM (short username oryx)
I20260807 11:30:21.304338 1619 webserver.cc:460] Webserver started at http://127.24.103.67:34059/ using document root <none> and password file <none>
I20260807 11:30:21.305090 1619 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:21.305189 1619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:21.305572 1619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:21.308756 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "5688689f848d4c818b3a1a0fdb6ec232"
format_stamp: "Formatted at 2026-08-07 11:30:21 on dist-test-slave-wl2h"
server_key: "69216f60081b6933bbb6de1035471978"
server_key_iv: "f838a20e84426ef9b35279f5c9dc879e"
server_key_version: "encryptionkey@0"
I20260807 11:30:21.309592 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "5688689f848d4c818b3a1a0fdb6ec232"
format_stamp: "Formatted at 2026-08-07 11:30:21 on dist-test-slave-wl2h"
server_key: "69216f60081b6933bbb6de1035471978"
server_key_iv: "f838a20e84426ef9b35279f5c9dc879e"
server_key_version: "encryptionkey@0"
I20260807 11:30:21.315464 1619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:30:21.319245 1634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:21.320782 1619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:30:21.320976 1619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "5688689f848d4c818b3a1a0fdb6ec232"
format_stamp: "Formatted at 2026-08-07 11:30:21 on dist-test-slave-wl2h"
server_key: "69216f60081b6933bbb6de1035471978"
server_key_iv: "f838a20e84426ef9b35279f5c9dc879e"
server_key_version: "encryptionkey@0"
I20260807 11:30:21.321204 1619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:21.340022 1619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:21.344816 1619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:21.345129 1619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:21.348453 1619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:21.348569 1619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:21.348722 1619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:21.348814 1619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:30:21 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102221, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:21.375845 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.349166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:47157 (local address 127.24.103.126:44615)
0807 11:30:21.349968 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:21.349974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:21.350840 (+ 866us) server_negotiation.cc:407] Connection header received
0807 11:30:21.353839 (+ 2999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:21.353846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:21.353963 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:21.354100 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:21.356279 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.357128 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.360882 (+ 3754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.361538 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.361722 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.368572 (+ 6850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:21.368579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:21.368582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:21.368644 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:21.371920 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:21.372860 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:21.372866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:21.372868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:21.373012 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:21.373521 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:21.373525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:21.373527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:21.373828 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:21.373995 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:21.374808 (+ 813us) server_negotiation.cc:299] Negotiation successful
0807 11:30:21.375113 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":545,"threads_started":1}
I20260807 11:30:21.377614 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.349787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44615 (local address 127.24.103.67:47157)
0807 11:30:21.350661 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:21.350710 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:21.353490 (+ 2780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:21.354386 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:21.354395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:21.354951 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:21.356053 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.356078 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.357338 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.357352 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.360556 (+ 3204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.360579 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.362116 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.362128 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.362552 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.363584 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:21.363624 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:21.368189 (+ 4565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:21.372165 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:21.372177 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:21.372200 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:21.372694 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:21.373140 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:21.373145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:21.373149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:21.373382 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:21.373968 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:21.373976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:21.374411 (+ 435us) client_negotiation.cc:769] Sending connection context
0807 11:30:21.374690 (+ 279us) client_negotiation.cc:240] Negotiation successful
0807 11:30:21.375602 (+ 912us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":389,"threads_started":1}
I20260807 11:30:21.412945 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.393056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:51035 (local address 127.24.103.126:44615)
0807 11:30:21.393311 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:21.393321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:21.394312 (+ 991us) server_negotiation.cc:407] Connection header received
0807 11:30:21.394583 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:21.394588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:21.394681 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:21.394805 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:21.396695 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.397994 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.401067 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.401587 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.401718 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.404871 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:21.404880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:21.404888 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:21.404941 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:21.408948 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:21.409978 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:21.409987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:21.409994 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:21.410214 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:21.410633 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:21.410641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:21.410647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:21.411161 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:21.411337 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:21.412124 (+ 787us) server_negotiation.cc:299] Negotiation successful
0807 11:30:21.412474 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:30:21.414541 1709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.393457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44615 (local address 127.24.103.67:51035)
0807 11:30:21.394124 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:21.394145 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:21.394404 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:21.394836 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:21.394840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:21.395693 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:21.396486 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.396511 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.398266 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.398275 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.400789 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.400814 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.401846 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.401855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.402111 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.402966 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:21.403000 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:21.404572 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:21.409109 (+ 4537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:21.409117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:21.409126 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:21.409799 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:21.410364 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:21.410372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:21.410375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:21.410504 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:21.411418 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:21.411426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:21.411720 (+ 294us) client_negotiation.cc:769] Sending connection context
0807 11:30:21.413246 (+ 1526us) client_negotiation.cc:240] Negotiation successful
0807 11:30:21.413744 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":371,"threads_started":1}
I20260807 11:30:21.417066 1619 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:43319
I20260807 11:30:21.417088 1768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:43319 every 8 connection(s)
I20260807 11:30:21.418798 1619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:30:21.421538 1769 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44615
I20260807 11:30:21.421772 1769 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:21.422546 1769 heartbeater.cc:511] Master 127.24.103.126:44615 requested a full tablet report, sending...
I20260807 11:30:21.424147 1227 ts_manager.cc:194] Registered new tserver with Master: 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67:43319)
I20260807 11:30:21.425094 1227 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:47157
I20260807 11:30:21.429041 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 1619
I20260807 11:30:21.429219 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:30:21.429697 24989 external_mini_cluster.cc:1428] Setting key 430b454a22314319919cf43a1f6d3352
I20260807 11:30:21.433066 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 11:30:21 dist-test-slave-wl2h krb5kdc[1180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102221, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 11:30:21 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:21 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:30:21.500751 1644 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:21.488960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.24.103.126:44615)
0807 11:30:21.489213 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:21.489221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:21.489475 (+ 254us) server_negotiation.cc:407] Connection header received
0807 11:30:21.489769 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:21.489774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:21.489843 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:21.489955 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:21.491450 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.492199 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.493480 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.493904 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.494048 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.500383 (+ 6335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39016: BlockingRecv error: recv got EOF from 127.0.0.1:39016 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 07 11:30:21 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102221, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:21.520139 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.504906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.24.103.126:44615)
0807 11:30:21.505158 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:21.505164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:21.505479 (+ 315us) server_negotiation.cc:407] Connection header received
0807 11:30:21.505723 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:21.505728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:21.505803 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:21.505911 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:21.507247 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.508094 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.509452 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.509885 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.509983 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.514135 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:21.514140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:21.514143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:21.514180 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:21.517041 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:21.517879 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:21.517883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:21.517885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:21.518014 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:21.518534 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:21.518537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:21.518540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:21.518880 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:21.519181 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:21.519562 (+ 381us) server_negotiation.cc:299] Negotiation successful
0807 11:30:21.519756 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:30:21.524802 1227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:30:21.532209 1227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:30:21.561537 1788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.546688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:43319 (local address 127.0.0.1:51020)
0807 11:30:21.550677 (+ 3989us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:21.550698 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:21.550812 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:21.553559 (+ 2747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:21.553565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:21.553591 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:21.554088 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.554104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.556407 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.556417 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.558624 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.558645 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.560522 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.560528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.560735 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.560793 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 11:30:21.560900 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 11:30:21.561102 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3846,"mutex_wait_us":218,"thread_start_us":399,"threads_started":1}
I20260807 11:30:21.562036 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.545914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:40989 (local address 127.0.0.1:57244)
0807 11:30:21.547640 (+ 1726us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:21.547706 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:21.547927 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:21.552196 (+ 4269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:21.552204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:21.552242 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:21.552753 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.552769 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.554884 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.554890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.559276 (+ 4386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.559291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.561144 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.561150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.561325 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.561358 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 11:30:21.561434 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 11:30:21.561576 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"mutex_wait_us":1728,"thread_start_us":4484,"threads_started":1}
I20260807 11:30:21.562660 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.546917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.24.103.67:43319)
0807 11:30:21.552961 (+ 6044us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:21.552969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:21.553003 (+ 34us) server_negotiation.cc:407] Connection header received
0807 11:30:21.553098 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:21.553106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:21.553325 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:21.554311 (+ 986us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:21.554622 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.556254 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.558787 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.560296 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.561814 (+ 1518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.562084 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:21.562153 (+ 69us) server_negotiation.cc:299] Negotiation successful
0807 11:30:21.562334 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5849,"thread_start_us":5587,"threads_started":1}
I20260807 11:30:21.563935 1789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.546156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.24.103.66:40989)
0807 11:30:21.551415 (+ 5259us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:21.551429 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:21.551467 (+ 38us) server_negotiation.cc:407] Connection header received
0807 11:30:21.551577 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:21.551586 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:21.551862 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:21.552989 (+ 1127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:21.553221 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.554731 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.559470 (+ 4739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.560888 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.563090 (+ 2202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.563306 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:21.563394 (+ 88us) server_negotiation.cc:299] Negotiation successful
0807 11:30:21.563595 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5050,"thread_start_us":4908,"threads_started":1}
I20260807 11:30:21.564435 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.551833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35005 (local address 127.0.0.1:39876)
0807 11:30:21.552539 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:21.552558 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:21.552789 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:21.553684 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:21.553689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:21.553721 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:21.554221 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.554230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.556022 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.556028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.560078 (+ 4050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.560095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.563572 (+ 3477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.563578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.563763 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.563795 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 11:30:21.563897 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 11:30:21.564058 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":1615,"thread_start_us":343,"threads_started":1}
I20260807 11:30:21.564435 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.551103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.24.103.65:35005)
0807 11:30:21.551940 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:21.551952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:21.552672 (+ 720us) server_negotiation.cc:407] Connection header received
0807 11:30:21.552997 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:21.553005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:21.553235 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:21.553402 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:21.554406 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.555812 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.560287 (+ 4475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.561742 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.561851 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.562156 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:21.563950 (+ 1794us) server_negotiation.cc:299] Negotiation successful
0807 11:30:21.564138 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":384,"threads_started":1}
I20260807 11:30:21.566560 1689 tablet_service.cc:1467] Processing CreateTablet for tablet 5765a8d81b5e47c39a22b9550a16ff08 (DEFAULT_TABLE table=test-table [id=339e1a3fde764887bbe7672eda7b4fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:21.567580 1536 tablet_service.cc:1467] Processing CreateTablet for tablet 5765a8d81b5e47c39a22b9550a16ff08 (DEFAULT_TABLE table=test-table [id=339e1a3fde764887bbe7672eda7b4fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:21.568141 1689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5765a8d81b5e47c39a22b9550a16ff08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:21.567988 1379 tablet_service.cc:1467] Processing CreateTablet for tablet 5765a8d81b5e47c39a22b9550a16ff08 (DEFAULT_TABLE table=test-table [id=339e1a3fde764887bbe7672eda7b4fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:21.569335 1379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5765a8d81b5e47c39a22b9550a16ff08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:21.570470 1536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5765a8d81b5e47c39a22b9550a16ff08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:21.584450 1794 tablet_bootstrap.cc:492] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: Bootstrap starting.
I20260807 11:30:21.587339 1795 tablet_bootstrap.cc:492] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: Bootstrap starting.
I20260807 11:30:21.590509 1794 tablet_bootstrap.cc:654] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:21.591497 1795 tablet_bootstrap.cc:654] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:21.591836 1796 tablet_bootstrap.cc:492] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: Bootstrap starting.
I20260807 11:30:21.592087 1794 log.cc:824] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:21.593079 1795 log.cc:824] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:21.595386 1794 tablet_bootstrap.cc:492] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: No bootstrap required, opened a new log
I20260807 11:30:21.595614 1796 tablet_bootstrap.cc:654] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:21.595698 1794 ts_tablet_manager.cc:1397] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260807 11:30:21.596670 1796 log.cc:824] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:21.597767 1795 tablet_bootstrap.cc:492] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: No bootstrap required, opened a new log
I20260807 11:30:21.598146 1795 ts_tablet_manager.cc:1397] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260807 11:30:21.600068 1796 tablet_bootstrap.cc:492] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: No bootstrap required, opened a new log
I20260807 11:30:21.600346 1796 ts_tablet_manager.cc:1397] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:30:21.606835 1794 raft_consensus.cc:348] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } }
I20260807 11:30:21.607420 1794 raft_consensus.cc:374] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:21.607568 1794 raft_consensus.cc:729] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5688689f848d4c818b3a1a0fdb6ec232, State: Initialized, Role: FOLLOWER
I20260807 11:30:21.608186 1796 raft_consensus.cc:348] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } }
I20260807 11:30:21.608278 1794 consensus_queue.cc:260] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } }
I20260807 11:30:21.608608 1796 raft_consensus.cc:374] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:21.608712 1796 raft_consensus.cc:729] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f47045eb9d47a2a8da13f1f5598e18, State: Initialized, Role: FOLLOWER
I20260807 11:30:21.608996 1795 raft_consensus.cc:348] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } }
I20260807 11:30:21.609344 1796 consensus_queue.cc:260] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } }
I20260807 11:30:21.609756 1769 heartbeater.cc:503] Master 127.24.103.126:44615 was elected leader, sending a full tablet report...
I20260807 11:30:21.610467 1794 ts_tablet_manager.cc:1428] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260807 11:30:21.610989 1615 heartbeater.cc:503] Master 127.24.103.126:44615 was elected leader, sending a full tablet report...
I20260807 11:30:21.611097 1796 ts_tablet_manager.cc:1428] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260807 11:30:21.611650 1795 raft_consensus.cc:374] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:21.612293 1795 raft_consensus.cc:729] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcea41ec2e954d4faeccf1bcabd3b4cc, State: Initialized, Role: FOLLOWER
I20260807 11:30:21.613075 1795 consensus_queue.cc:260] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } }
I20260807 11:30:21.616235 1795 ts_tablet_manager.cc:1428] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: Time spent starting tablet: real 0.018s user 0.013s sys 0.000s
W20260807 11:30:21.671263 1770 tablet.cc:2369] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:30:21.752183 1461 tablet.cc:2369] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:30:21.822235 1801 raft_consensus.cc:482] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:30:21.822547 1801 raft_consensus.cc:504] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } }
I20260807 11:30:21.824404 1801 leader_election.cc:288] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989), bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65:35005)
I20260807 11:30:21.831956 1789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.824905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:50859 (local address 127.24.103.66:40989)
0807 11:30:21.825180 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:21.825188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:21.825213 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:30:21.825312 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:21.825320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:21.825405 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:21.825505 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:21.826067 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.827119 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.829487 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.830768 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.830981 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.831086 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:21.831542 (+ 456us) server_negotiation.cc:299] Negotiation successful
0807 11:30:21.831696 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260807 11:30:21.832024 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.825447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:56759 (local address 127.24.103.65:35005)
0807 11:30:21.825712 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:21.825718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:21.825741 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:30:21.825825 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:21.825829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:21.825888 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:21.825962 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:21.826610 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.827523 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.829805 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.831074 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.831228 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.831278 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:21.831662 (+ 384us) server_negotiation.cc:299] Negotiation successful
0807 11:30:21.831794 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 11:30:21.832082 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.825030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35005 (local address 127.24.103.67:56759)
0807 11:30:21.825589 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:21.825607 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:21.825708 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:21.826006 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:21.826010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:21.826049 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:21.826421 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.826430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.827650 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.827655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.829641 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.829651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.831323 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.831330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.831501 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.831529 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 11:30:21.831606 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 11:30:21.831731 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":311,"threads_started":1}
I20260807 11:30:21.832038 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.824640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:40989 (local address 127.24.103.67:50859)
0807 11:30:21.824824 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:21.824843 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:21.825001 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:21.825635 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:21.825643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:21.825679 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:21.825895 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.825904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.827248 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.827252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.829297 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:21.829315 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.830992 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.830997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:21.831335 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.831372 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 11:30:21.831498 (+ 126us) client_negotiation.cc:240] Negotiation successful
0807 11:30:21.831656 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260807 11:30:21.833300 1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5765a8d81b5e47c39a22b9550a16ff08" candidate_uuid: "5688689f848d4c818b3a1a0fdb6ec232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" is_pre_election: true
I20260807 11:30:21.833856 1401 raft_consensus.cc:2454] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5688689f848d4c818b3a1a0fdb6ec232 in term 0.
I20260807 11:30:21.833792 1561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5765a8d81b5e47c39a22b9550a16ff08" candidate_uuid: "5688689f848d4c818b3a1a0fdb6ec232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f47045eb9d47a2a8da13f1f5598e18" is_pre_election: true
I20260807 11:30:21.834337 1561 raft_consensus.cc:2454] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5688689f848d4c818b3a1a0fdb6ec232 in term 0.
I20260807 11:30:21.834641 1635 leader_election.cc:302] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5688689f848d4c818b3a1a0fdb6ec232, bcea41ec2e954d4faeccf1bcabd3b4cc; no voters:
I20260807 11:30:21.835224 1801 raft_consensus.cc:2781] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:30:21.835417 1801 raft_consensus.cc:482] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:30:21.835531 1801 raft_consensus.cc:3037] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:21.837872 1801 raft_consensus.cc:504] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } }
I20260807 11:30:21.838479 1801 leader_election.cc:288] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [CANDIDATE]: Term 1 election: Requested vote from peers 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989), bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65:35005)
I20260807 11:30:21.839133 1561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5765a8d81b5e47c39a22b9550a16ff08" candidate_uuid: "5688689f848d4c818b3a1a0fdb6ec232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f47045eb9d47a2a8da13f1f5598e18"
I20260807 11:30:21.839150 1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5765a8d81b5e47c39a22b9550a16ff08" candidate_uuid: "5688689f848d4c818b3a1a0fdb6ec232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc"
I20260807 11:30:21.839354 1561 raft_consensus.cc:3037] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:21.839385 1401 raft_consensus.cc:3037] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:30:21.841763 1616 tablet.cc:2369] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:30:21.842114 1401 raft_consensus.cc:2454] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5688689f848d4c818b3a1a0fdb6ec232 in term 1.
I20260807 11:30:21.842893 1561 raft_consensus.cc:2454] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5688689f848d4c818b3a1a0fdb6ec232 in term 1.
I20260807 11:30:21.843250 1635 leader_election.cc:302] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5688689f848d4c818b3a1a0fdb6ec232, bcea41ec2e954d4faeccf1bcabd3b4cc; no voters:
I20260807 11:30:21.843586 1801 raft_consensus.cc:2781] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:21.844224 1801 raft_consensus.cc:686] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 LEADER]: Becoming Leader. State: Replica: 5688689f848d4c818b3a1a0fdb6ec232, State: Running, Role: LEADER
I20260807 11:30:21.845206 1801 consensus_queue.cc:237] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } }
I20260807 11:30:21.850729 1227 catalog_manager.cc:5495] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "5688689f848d4c818b3a1a0fdb6ec232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } health_report { overall_health: UNKNOWN } } }
I20260807 11:30:21.930557 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:21.924905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.24.103.67:43319)
0807 11:30:21.925199 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:21.925206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:21.925252 (+ 46us) server_negotiation.cc:407] Connection header received
0807 11:30:21.925554 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:21.925559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:21.925671 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:21.925840 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:30:21.926229 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.926928 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.928271 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:21.928681 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:21.928819 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:21.929310 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:30:21.929872 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:30:21.930031 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:21.930141 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 11:30:21.930231 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260807 11:30:21.939518 1561 raft_consensus.cc:1261] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Refusing update from remote peer 5688689f848d4c818b3a1a0fdb6ec232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:21.939648 1401 raft_consensus.cc:1261] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Refusing update from remote peer 5688689f848d4c818b3a1a0fdb6ec232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:21.940608 1805 consensus_queue.cc:1035] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:21.941036 1801 consensus_queue.cc:1035] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:21.959796 1810 mvcc.cc:205] Tried to move back new op lower bound from 7315874701052878848 to 7315874700683026432. Current Snapshot: MvccSnapshot[applied={T|T < 7315874701052878848}]
I20260807 11:30:21.972559 1228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:21.972893 1228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:21.978615 1228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949: Sending DeleteTablet for 3 replicas of tablet 5765a8d81b5e47c39a22b9550a16ff08
I20260807 11:30:21.980206 1689 tablet_service.cc:1514] Processing DeleteTablet for tablet 5765a8d81b5e47c39a22b9550a16ff08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:21 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:51020
I20260807 11:30:21.980243 1536 tablet_service.cc:1514] Processing DeleteTablet for tablet 5765a8d81b5e47c39a22b9550a16ff08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:21 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:57244
I20260807 11:30:21.980677 1379 tablet_service.cc:1514] Processing DeleteTablet for tablet 5765a8d81b5e47c39a22b9550a16ff08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:21 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:39876
I20260807 11:30:21.983050 1228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:30:21.983872 1228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:30:21.994361 1820 tablet_replica.cc:330] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: stopping tablet replica
I20260807 11:30:21.995537 1820 raft_consensus.cc:2229] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:21.996399 1688 tablet_service.cc:1467] Processing CreateTablet for tablet 6344e7a9b4964ac6a1be44cfa0ebc465 (DEFAULT_TABLE table=test-table [id=aa4b747bd4924758985d8bda505f05e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:21.996690 1820 raft_consensus.cc:2258] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:21.997057 1688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6344e7a9b4964ac6a1be44cfa0ebc465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:22.000691 1820 ts_tablet_manager.cc:1905] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:22.002288 1818 tablet_replica.cc:330] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: stopping tablet replica
I20260807 11:30:22.002390 1819 tablet_replica.cc:330] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: stopping tablet replica
I20260807 11:30:22.002897 1819 raft_consensus.cc:2229] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:22.003186 1379 tablet_service.cc:1467] Processing CreateTablet for tablet 6344e7a9b4964ac6a1be44cfa0ebc465 (DEFAULT_TABLE table=test-table [id=aa4b747bd4924758985d8bda505f05e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:22.003575 1819 raft_consensus.cc:2258] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:22.003680 1379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6344e7a9b4964ac6a1be44cfa0ebc465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:22.004770 1536 tablet_service.cc:1467] Processing CreateTablet for tablet 6344e7a9b4964ac6a1be44cfa0ebc465 (DEFAULT_TABLE table=test-table [id=aa4b747bd4924758985d8bda505f05e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:22.005301 1536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6344e7a9b4964ac6a1be44cfa0ebc465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:22.006788 1819 ts_tablet_manager.cc:1905] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:22.008502 1818 raft_consensus.cc:2229] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:30:22.009330 1818 raft_consensus.cc:2258] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:22.010262 1795 tablet_bootstrap.cc:492] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc: Bootstrap starting.
I20260807 11:30:22.012094 1795 tablet_bootstrap.cc:654] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:22.013217 1794 tablet_bootstrap.cc:492] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232: Bootstrap starting.
I20260807 11:30:22.015030 1794 tablet_bootstrap.cc:654] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:22.015295 1796 tablet_bootstrap.cc:492] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18: Bootstrap starting.
I20260807 11:30:22.015734 1820 ts_tablet_manager.cc:1918] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:22.015887 1820 log.cc:1196] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/5765a8d81b5e47c39a22b9550a16ff08
I20260807 11:30:22.016571 1820 ts_tablet_manager.cc:1939] T 5765a8d81b5e47c39a22b9550a16ff08 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting consensus metadata
I20260807 11:30:22.016959 1796 tablet_bootstrap.cc:654] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:22.018947 1211 catalog_manager.cc:4841] TS bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65:35005): tablet 5765a8d81b5e47c39a22b9550a16ff08 (table test-table [id=339e1a3fde764887bbe7672eda7b4fbe]) successfully deleted
I20260807 11:30:22.019974 1818 ts_tablet_manager.cc:1905] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:22.025377 1819 ts_tablet_manager.cc:1918] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:22.026106 1795 tablet_bootstrap.cc:492] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc: No bootstrap required, opened a new log
I20260807 11:30:22.026279 1795 ts_tablet_manager.cc:1397] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.009s
I20260807 11:30:22.026305 1819 log.cc:1196] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/5765a8d81b5e47c39a22b9550a16ff08
I20260807 11:30:22.027179 1795 raft_consensus.cc:348] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.027431 1795 raft_consensus.cc:374] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:22.027536 1795 raft_consensus.cc:729] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcea41ec2e954d4faeccf1bcabd3b4cc, State: Initialized, Role: FOLLOWER
I20260807 11:30:22.027828 1795 consensus_queue.cc:260] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.028440 1795 ts_tablet_manager.cc:1428] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260807 11:30:22.028569 1796 tablet_bootstrap.cc:492] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18: No bootstrap required, opened a new log
I20260807 11:30:22.028769 1796 ts_tablet_manager.cc:1397] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260807 11:30:22.029486 1794 tablet_bootstrap.cc:492] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232: No bootstrap required, opened a new log
I20260807 11:30:22.029708 1794 ts_tablet_manager.cc:1397] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260807 11:30:22.029726 1796 raft_consensus.cc:348] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.029942 1796 raft_consensus.cc:374] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:22.030047 1796 raft_consensus.cc:729] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f47045eb9d47a2a8da13f1f5598e18, State: Initialized, Role: FOLLOWER
I20260807 11:30:22.030354 1796 consensus_queue.cc:260] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.030834 1794 raft_consensus.cc:348] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.031144 1794 raft_consensus.cc:374] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:22.031306 1794 raft_consensus.cc:729] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5688689f848d4c818b3a1a0fdb6ec232, State: Initialized, Role: FOLLOWER
I20260807 11:30:22.031643 1796 ts_tablet_manager.cc:1428] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 11:30:22.031898 1819 ts_tablet_manager.cc:1939] T 5765a8d81b5e47c39a22b9550a16ff08 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting consensus metadata
I20260807 11:30:22.031687 1794 consensus_queue.cc:260] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.032377 1794 ts_tablet_manager.cc:1428] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260807 11:30:22.034432 1211 catalog_manager.cc:4841] TS 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989): tablet 5765a8d81b5e47c39a22b9550a16ff08 (table test-table [id=339e1a3fde764887bbe7672eda7b4fbe]) successfully deleted
I20260807 11:30:22.035495 1818 ts_tablet_manager.cc:1918] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:22.035718 1818 log.cc:1196] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/5765a8d81b5e47c39a22b9550a16ff08
I20260807 11:30:22.036441 1818 ts_tablet_manager.cc:1939] T 5765a8d81b5e47c39a22b9550a16ff08 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting consensus metadata
I20260807 11:30:22.038041 1212 catalog_manager.cc:4841] TS 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67:43319): tablet 5765a8d81b5e47c39a22b9550a16ff08 (table test-table [id=339e1a3fde764887bbe7672eda7b4fbe]) successfully deleted
I20260807 11:30:22.076268 1803 raft_consensus.cc:482] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:30:22.076537 1803 raft_consensus.cc:504] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.078543 1803 leader_election.cc:288] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989), 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67:43319)
I20260807 11:30:22.088148 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.080101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:32921 (local address 127.24.103.67:43319)
0807 11:30:22.080275 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:22.080281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:22.081134 (+ 853us) server_negotiation.cc:407] Connection header received
0807 11:30:22.081419 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:22.081427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:22.081508 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:22.081570 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:22.082423 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.083537 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.085484 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.086796 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.086973 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.087087 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:22.087682 (+ 595us) server_negotiation.cc:299] Negotiation successful
0807 11:30:22.087824 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:30:22.088150 1825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.079243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:43319 (local address 127.24.103.65:32921)
0807 11:30:22.081031 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:22.081050 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:22.081257 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:22.081817 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:22.081822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:22.081850 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:22.082216 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.082228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.083695 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.083700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.085315 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.085328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.087273 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.087279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.087500 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.087526 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 11:30:22.087638 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 11:30:22.087792 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1657,"thread_start_us":1068,"threads_started":1}
I20260807 11:30:22.088259 1789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.079750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:57867 (local address 127.24.103.66:40989)
0807 11:30:22.079943 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:22.079950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:22.080853 (+ 903us) server_negotiation.cc:407] Connection header received
0807 11:30:22.081161 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:22.081165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:22.081226 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:22.081327 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:22.082007 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.083157 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.085828 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.087148 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.087312 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.087395 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:22.087817 (+ 422us) server_negotiation.cc:299] Negotiation successful
0807 11:30:22.087960 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 11:30:22.089274 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.078956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:40989 (local address 127.24.103.65:57867)
0807 11:30:22.080726 (+ 1770us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:22.080751 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:22.080975 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:22.081380 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:22.081386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:22.081410 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:22.081847 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.081859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.083306 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.083314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.085637 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.085654 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.087402 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.087409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.087587 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.087623 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 11:30:22.088577 (+ 954us) client_negotiation.cc:240] Negotiation successful
0807 11:30:22.088892 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1575,"mutex_wait_us":4,"thread_start_us":1717,"threads_started":1}
I20260807 11:30:22.089391 1715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6344e7a9b4964ac6a1be44cfa0ebc465" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5688689f848d4c818b3a1a0fdb6ec232" is_pre_election: true
I20260807 11:30:22.089901 1561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6344e7a9b4964ac6a1be44cfa0ebc465" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f47045eb9d47a2a8da13f1f5598e18" is_pre_election: true
I20260807 11:30:22.089951 1715 raft_consensus.cc:2454] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 0.
I20260807 11:30:22.090168 1561 raft_consensus.cc:2454] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 0.
I20260807 11:30:22.090751 1325 leader_election.cc:302] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5688689f848d4c818b3a1a0fdb6ec232, 92f47045eb9d47a2a8da13f1f5598e18, bcea41ec2e954d4faeccf1bcabd3b4cc; no voters:
I20260807 11:30:22.091284 1803 raft_consensus.cc:2781] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:30:22.091427 1803 raft_consensus.cc:482] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:30:22.091552 1803 raft_consensus.cc:3037] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:22.093910 1803 raft_consensus.cc:504] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.094564 1803 leader_election.cc:288] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 election: Requested vote from peers 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989), 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67:43319)
I20260807 11:30:22.095120 1715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6344e7a9b4964ac6a1be44cfa0ebc465" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5688689f848d4c818b3a1a0fdb6ec232"
I20260807 11:30:22.095115 1561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6344e7a9b4964ac6a1be44cfa0ebc465" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f47045eb9d47a2a8da13f1f5598e18"
I20260807 11:30:22.095350 1715 raft_consensus.cc:3037] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:22.095343 1561 raft_consensus.cc:3037] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:22.098742 1715 raft_consensus.cc:2454] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 1.
I20260807 11:30:22.099098 1561 raft_consensus.cc:2454] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 1.
I20260807 11:30:22.099246 1326 leader_election.cc:302] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5688689f848d4c818b3a1a0fdb6ec232, bcea41ec2e954d4faeccf1bcabd3b4cc; no voters:
I20260807 11:30:22.099582 1803 raft_consensus.cc:2781] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:22.100124 1803 raft_consensus.cc:686] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 LEADER]: Becoming Leader. State: Replica: bcea41ec2e954d4faeccf1bcabd3b4cc, State: Running, Role: LEADER
I20260807 11:30:22.100519 1803 consensus_queue.cc:237] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.104259 1228 catalog_manager.cc:5495] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc reported cstate change: term changed from 0 to 1, leader changed from <none> to bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65). New cstate: current_term: 1 leader_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } health_report { overall_health: UNKNOWN } } }
Aug 07 11:30:22 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102219, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.126@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:22.171299 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.153831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44615 (local address 127.0.0.1:39038)
0807 11:30:22.154844 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:22.154862 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:22.155001 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:22.155464 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:22.155468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:22.156250 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:22.156464 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.156474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.158048 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.158053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.159231 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.159245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.160341 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.160346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.160481 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.161418 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:22.161443 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:22.163818 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:22.168313 (+ 4495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:22.168328 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:22.168336 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:22.168840 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:22.169297 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:22.169301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:22.169303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:22.169460 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:22.170170 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:22.170177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:22.170385 (+ 208us) client_negotiation.cc:769] Sending connection context
0807 11:30:22.170538 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 11:30:22.170800 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":677,"threads_started":1}
I20260807 11:30:22.171581 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.153852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.24.103.126:44615)
0807 11:30:22.154613 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:22.154620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:22.154965 (+ 345us) server_negotiation.cc:407] Connection header received
0807 11:30:22.155198 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:22.155204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:22.155283 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:22.155409 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:22.156618 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.157870 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.159383 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.160148 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.160289 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.164075 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:22.164083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:22.164088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:22.164130 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:22.168112 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:22.168990 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:22.168995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:22.168999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:22.169165 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:22.169585 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:22.169590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:22.169594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:22.169995 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:22.170335 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:22.170750 (+ 415us) server_negotiation.cc:299] Negotiation successful
0807 11:30:22.171060 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":399,"threads_started":1}
I20260807 11:30:22.177902 1228 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:39038:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 11:30:22.179096 1228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:30:22.186810 1379 tablet_service.cc:1467] Processing CreateTablet for tablet cb64d842ac2240d09f9c8a6741b7088a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0572103b8768431aa7a1e7a7e6018600]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:30:22.186950 1536 tablet_service.cc:1467] Processing CreateTablet for tablet cb64d842ac2240d09f9c8a6741b7088a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0572103b8768431aa7a1e7a7e6018600]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:30:22.187572 1379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb64d842ac2240d09f9c8a6741b7088a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:22.187408 1688 tablet_service.cc:1467] Processing CreateTablet for tablet cb64d842ac2240d09f9c8a6741b7088a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0572103b8768431aa7a1e7a7e6018600]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 11:30:22.187625 1536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb64d842ac2240d09f9c8a6741b7088a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:22.187960 1688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb64d842ac2240d09f9c8a6741b7088a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:22.195308 1796 tablet_bootstrap.cc:492] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18: Bootstrap starting.
I20260807 11:30:22.195518 1795 tablet_bootstrap.cc:492] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc: Bootstrap starting.
I20260807 11:30:22.196794 1794 tablet_bootstrap.cc:492] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: Bootstrap starting.
I20260807 11:30:22.197165 1795 tablet_bootstrap.cc:654] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:22.197165 1796 tablet_bootstrap.cc:654] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:22.198586 1794 tablet_bootstrap.cc:654] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:22.209148 1795 tablet_bootstrap.cc:492] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc: No bootstrap required, opened a new log
I20260807 11:30:22.209503 1795 ts_tablet_manager.cc:1397] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260807 11:30:22.210626 1795 raft_consensus.cc:348] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.211146 1795 raft_consensus.cc:374] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:22.211300 1795 raft_consensus.cc:729] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcea41ec2e954d4faeccf1bcabd3b4cc, State: Initialized, Role: FOLLOWER
I20260807 11:30:22.211664 1795 consensus_queue.cc:260] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.212275 1803 tablet_replica.cc:439] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } }
I20260807 11:30:22.212600 1803 tablet_replica.cc:442] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:30:22.212795 1796 tablet_bootstrap.cc:492] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18: No bootstrap required, opened a new log
I20260807 11:30:22.212968 1796 ts_tablet_manager.cc:1397] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20260807 11:30:22.213932 1796 raft_consensus.cc:348] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.214249 1796 raft_consensus.cc:374] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:22.214340 1796 raft_consensus.cc:729] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f47045eb9d47a2a8da13f1f5598e18, State: Initialized, Role: FOLLOWER
I20260807 11:30:22.214685 1796 consensus_queue.cc:260] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.215175 1802 tablet_replica.cc:439] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } }
I20260807 11:30:22.215551 1802 tablet_replica.cc:442] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:30:22.215724 1795 ts_tablet_manager.cc:1428] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260807 11:30:22.216622 1794 tablet_bootstrap.cc:492] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: No bootstrap required, opened a new log
I20260807 11:30:22.216814 1794 ts_tablet_manager.cc:1397] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.006s
I20260807 11:30:22.217665 1794 raft_consensus.cc:348] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.217911 1794 raft_consensus.cc:374] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:22.217985 1794 raft_consensus.cc:729] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5688689f848d4c818b3a1a0fdb6ec232, State: Initialized, Role: FOLLOWER
I20260807 11:30:22.218343 1796 ts_tablet_manager.cc:1428] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260807 11:30:22.218236 1794 consensus_queue.cc:260] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.218694 1813 tablet_replica.cc:439] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } }
I20260807 11:30:22.219115 1813 tablet_replica.cc:442] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:30:22.220110 1794 ts_tablet_manager.cc:1428] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260807 11:30:22.328461 1813 raft_consensus.cc:482] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:30:22.328794 1813 raft_consensus.cc:504] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.329869 1813 leader_election.cc:288] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65:35005), 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989)
I20260807 11:30:22.330621 1561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb64d842ac2240d09f9c8a6741b7088a" candidate_uuid: "5688689f848d4c818b3a1a0fdb6ec232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f47045eb9d47a2a8da13f1f5598e18" is_pre_election: true
I20260807 11:30:22.330657 1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb64d842ac2240d09f9c8a6741b7088a" candidate_uuid: "5688689f848d4c818b3a1a0fdb6ec232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" is_pre_election: true
I20260807 11:30:22.330992 1561 raft_consensus.cc:2454] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5688689f848d4c818b3a1a0fdb6ec232 in term 0.
I20260807 11:30:22.331063 1401 raft_consensus.cc:2454] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5688689f848d4c818b3a1a0fdb6ec232 in term 0.
I20260807 11:30:22.331534 1635 leader_election.cc:302] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5688689f848d4c818b3a1a0fdb6ec232, 92f47045eb9d47a2a8da13f1f5598e18; no voters:
I20260807 11:30:22.331928 1813 raft_consensus.cc:2781] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:30:22.332095 1813 raft_consensus.cc:482] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:30:22.332234 1813 raft_consensus.cc:3037] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:22.334674 1813 raft_consensus.cc:504] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.335320 1813 leader_election.cc:288] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [CANDIDATE]: Term 1 election: Requested vote from peers bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65:35005), 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989)
I20260807 11:30:22.335973 1561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb64d842ac2240d09f9c8a6741b7088a" candidate_uuid: "5688689f848d4c818b3a1a0fdb6ec232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f47045eb9d47a2a8da13f1f5598e18"
I20260807 11:30:22.336242 1561 raft_consensus.cc:3037] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:22.336243 1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb64d842ac2240d09f9c8a6741b7088a" candidate_uuid: "5688689f848d4c818b3a1a0fdb6ec232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc"
I20260807 11:30:22.336467 1401 raft_consensus.cc:3037] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:22.339293 1561 raft_consensus.cc:2454] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5688689f848d4c818b3a1a0fdb6ec232 in term 1.
I20260807 11:30:22.339380 1401 raft_consensus.cc:2454] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5688689f848d4c818b3a1a0fdb6ec232 in term 1.
I20260807 11:30:22.340034 1635 leader_election.cc:302] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5688689f848d4c818b3a1a0fdb6ec232, bcea41ec2e954d4faeccf1bcabd3b4cc; no voters:
I20260807 11:30:22.340351 1813 raft_consensus.cc:2781] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:22.340560 1813 raft_consensus.cc:686] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 LEADER]: Becoming Leader. State: Replica: 5688689f848d4c818b3a1a0fdb6ec232, State: Running, Role: LEADER
I20260807 11:30:22.340945 1813 consensus_queue.cc:237] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.342892 1805 tablet_replica.cc:439] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: TxnStatusTablet state changed. Reason: New leader 5688689f848d4c818b3a1a0fdb6ec232. Latest consensus state: current_term: 1 leader_uuid: "5688689f848d4c818b3a1a0fdb6ec232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } }
I20260807 11:30:22.343907 1805 tablet_replica.cc:442] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:30:22.344532 1844 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 11:30:22.344848 1844 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 11:30:22.345237 1226 catalog_manager.cc:5495] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "5688689f848d4c818b3a1a0fdb6ec232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } health_report { overall_health: HEALTHY } } }
I20260807 11:30:22.371716 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.363788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.24.103.67:43319)
0807 11:30:22.363990 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:22.363995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:22.364116 (+ 121us) server_negotiation.cc:407] Connection header received
0807 11:30:22.364430 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:22.364435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:22.364507 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:22.364598 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:30:22.365113 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.365910 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.368166 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.369411 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.369571 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.370615 (+ 1044us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:30:22.370790 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:30:22.370907 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:22.371232 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 11:30:22.371336 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:30:22.371789 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.363669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:43319 (local address 127.0.0.1:51038)
0807 11:30:22.363975 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:22.364000 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:22.364177 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:22.364678 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:22.364685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:22.364700 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:30:22.364918 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.364931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.366077 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.366084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.367969 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.367987 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.369739 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.369795 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.370154 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.370266 (+ 112us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:30:22.370975 (+ 709us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:30:22.371004 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 11:30:22.371169 (+ 165us) client_negotiation.cc:240] Negotiation successful
0807 11:30:22.371382 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260807 11:30:22.377975 1401 raft_consensus.cc:1261] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Refusing update from remote peer 5688689f848d4c818b3a1a0fdb6ec232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:22.378002 1561 raft_consensus.cc:1261] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Refusing update from remote peer 5688689f848d4c818b3a1a0fdb6ec232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:22.378701 1805 consensus_queue.cc:1035] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:22.379078 1813 consensus_queue.cc:1035] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:22.383234 1802 tablet_replica.cc:439] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18: TxnStatusTablet state changed. Reason: New leader 5688689f848d4c818b3a1a0fdb6ec232. Latest consensus state: current_term: 1 leader_uuid: "5688689f848d4c818b3a1a0fdb6ec232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } }
I20260807 11:30:22.383571 1802 tablet_replica.cc:442] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:30:22.386689 1808 mvcc.cc:205] Tried to move back new op lower bound from 7315874702851108864 to 7315874702712061952. Current Snapshot: MvccSnapshot[applied={T|T < 7315874702851108864}]
I20260807 11:30:22.388428 1802 tablet_replica.cc:439] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5688689f848d4c818b3a1a0fdb6ec232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } }
I20260807 11:30:22.388680 1802 tablet_replica.cc:442] T cb64d842ac2240d09f9c8a6741b7088a P 92f47045eb9d47a2a8da13f1f5598e18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:30:22.387271 1805 tablet_replica.cc:439] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5688689f848d4c818b3a1a0fdb6ec232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } }
I20260807 11:30:22.389952 1805 tablet_replica.cc:442] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:30:22.395792 1803 tablet_replica.cc:439] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc: TxnStatusTablet state changed. Reason: New leader 5688689f848d4c818b3a1a0fdb6ec232. Latest consensus state: current_term: 1 leader_uuid: "5688689f848d4c818b3a1a0fdb6ec232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } }
I20260807 11:30:22.396119 1803 tablet_replica.cc:442] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:30:22.398131 1803 tablet_replica.cc:439] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5688689f848d4c818b3a1a0fdb6ec232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } }
I20260807 11:30:22.398427 1803 tablet_replica.cc:442] T cb64d842ac2240d09f9c8a6741b7088a P bcea41ec2e954d4faeccf1bcabd3b4cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 11:30:22.398921 1805 tablet_replica.cc:439] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5688689f848d4c818b3a1a0fdb6ec232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } } }
I20260807 11:30:22.399463 1805 tablet_replica.cc:442] T cb64d842ac2240d09f9c8a6741b7088a P 5688689f848d4c818b3a1a0fdb6ec232: This TxnStatusTablet replica's current role is: LEADER
I20260807 11:30:22.401633 1809 mvcc.cc:205] Tried to move back new op lower bound from 7315874702851108864 to 7315874702712061952. Current Snapshot: MvccSnapshot[applied={T|T < 7315874702851108864 or (T in {7315874702851108864})}]
I20260807 11:30:22.418156 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.399413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.24.103.65:35005)
0807 11:30:22.409118 (+ 9705us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:22.409124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:22.409155 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:30:22.409274 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:22.409279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:22.409409 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:22.409691 (+ 282us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:30:22.410328 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.411765 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.414209 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.415078 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.415890 (+ 812us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.416088 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:30:22.416916 (+ 828us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:30:22.417619 (+ 703us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:22.417678 (+ 59us) server_negotiation.cc:299] Negotiation successful
0807 11:30:22.417773 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9530,"thread_start_us":9382,"threads_started":1}
I20260807 11:30:22.439240 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.431406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:50693 (local address 127.24.103.67:43319)
0807 11:30:22.431600 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:22.431606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:22.431806 (+ 200us) server_negotiation.cc:407] Connection header received
0807 11:30:22.432116 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:22.432121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:22.432207 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:22.432307 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:30:22.432830 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.433718 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.436971 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.437657 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.437838 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.438318 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:30:22.438492 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:30:22.438593 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:22.438756 (+ 163us) server_negotiation.cc:299] Negotiation successful
0807 11:30:22.438843 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:30:22.439339 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.430921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:43319 (local address 127.24.103.65:50693)
0807 11:30:22.431701 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:22.431724 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:22.431944 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:22.432330 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:22.432335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:22.432346 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:30:22.432662 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.432672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.433864 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.433870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.434965 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.434976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.437876 (+ 2900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.437884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.438044 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.438104 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:30:22.438599 (+ 495us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:30:22.438614 (+ 15us) client_negotiation.cc:769] Sending connection context
0807 11:30:22.438722 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 11:30:22.438873 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":698,"threads_started":1}
I20260807 11:30:22.464274 1561 raft_consensus.cc:1261] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Refusing update from remote peer bcea41ec2e954d4faeccf1bcabd3b4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:22.464320 1715 raft_consensus.cc:1261] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Refusing update from remote peer bcea41ec2e954d4faeccf1bcabd3b4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:22.465286 1829 consensus_queue.cc:1035] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:22.465739 1803 consensus_queue.cc:1035] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:22.534359 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.519971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:43839 (local address 127.24.103.65:35005)
0807 11:30:22.520343 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:22.520373 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:22.527741 (+ 7368us) server_negotiation.cc:407] Connection header received
0807 11:30:22.528045 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:22.528054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:22.528221 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:22.528332 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:30:22.529078 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.529928 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.531829 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.532557 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.532731 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.533291 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:30:22.533487 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:30:22.533600 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:22.533839 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 11:30:22.533947 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260807 11:30:22.534518 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.519913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35005 (local address 127.24.103.67:43839)
0807 11:30:22.527615 (+ 7702us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:22.527637 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:22.527813 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:22.528381 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:22.528385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:22.528398 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 11:30:22.528869 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.528887 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.530059 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.530064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.531600 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:22.531618 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.532820 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.532828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:22.532986 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.533059 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 11:30:22.533668 (+ 609us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 11:30:22.533681 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 11:30:22.533780 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 11:30:22.533956 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7471,"thread_start_us":7297,"threads_started":1}
I20260807 11:30:22.605533 1226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:22.605827 1226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:22.610086 1226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949: Sending DeleteTablet for 3 replicas of tablet 6344e7a9b4964ac6a1be44cfa0ebc465
I20260807 11:30:22.611915 1536 tablet_service.cc:1514] Processing DeleteTablet for tablet 6344e7a9b4964ac6a1be44cfa0ebc465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:22 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:57244
I20260807 11:30:22.612437 1688 tablet_service.cc:1514] Processing DeleteTablet for tablet 6344e7a9b4964ac6a1be44cfa0ebc465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:22 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:51020
I20260807 11:30:22.614043 1379 tablet_service.cc:1514] Processing DeleteTablet for tablet 6344e7a9b4964ac6a1be44cfa0ebc465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:22 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:39876
I20260807 11:30:22.625962 1862 tablet_replica.cc:330] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18: stopping tablet replica
I20260807 11:30:22.626214 1862 raft_consensus.cc:2229] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:22.626513 1862 raft_consensus.cc:2258] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:22.627697 1862 ts_tablet_manager.cc:1905] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:22.635821 1862 ts_tablet_manager.cc:1918] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:30:22.635996 1862 log.cc:1196] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/6344e7a9b4964ac6a1be44cfa0ebc465
I20260807 11:30:22.636490 1862 ts_tablet_manager.cc:1939] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting consensus metadata
I20260807 11:30:22.637907 1211 catalog_manager.cc:4841] TS 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989): tablet 6344e7a9b4964ac6a1be44cfa0ebc465 (table test-table [id=aa4b747bd4924758985d8bda505f05e2]) successfully deleted
I20260807 11:30:22.657119 1864 tablet_replica.cc:330] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc: stopping tablet replica
I20260807 11:30:22.657394 1864 raft_consensus.cc:2229] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:30:22.658071 1864 raft_consensus.cc:2258] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:22.658821 1863 tablet_replica.cc:330] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232: stopping tablet replica
I20260807 11:30:22.659081 1863 raft_consensus.cc:2229] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:22.659404 1863 raft_consensus.cc:2258] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:22.660848 1863 ts_tablet_manager.cc:1905] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:22.664099 1864 ts_tablet_manager.cc:1905] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:22.693620 1863 ts_tablet_manager.cc:1918] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:30:22.693805 1863 log.cc:1196] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/6344e7a9b4964ac6a1be44cfa0ebc465
I20260807 11:30:22.694340 1863 ts_tablet_manager.cc:1939] T 6344e7a9b4964ac6a1be44cfa0ebc465 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting consensus metadata
I20260807 11:30:22.695748 1212 catalog_manager.cc:4841] TS 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67:43319): tablet 6344e7a9b4964ac6a1be44cfa0ebc465 (table test-table [id=aa4b747bd4924758985d8bda505f05e2]) successfully deleted
I20260807 11:30:22.697568 1864 ts_tablet_manager.cc:1918] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:30:22.697737 1864 log.cc:1196] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/6344e7a9b4964ac6a1be44cfa0ebc465
I20260807 11:30:22.698294 1864 ts_tablet_manager.cc:1939] T 6344e7a9b4964ac6a1be44cfa0ebc465 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting consensus metadata
I20260807 11:30:22.699577 1211 catalog_manager.cc:4841] TS bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65:35005): tablet 6344e7a9b4964ac6a1be44cfa0ebc465 (table test-table [id=aa4b747bd4924758985d8bda505f05e2]) successfully deleted
Aug 07 11:30:22 dist-test-slave-wl2h krb5kdc[1180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260807 11:30:22.740768 24989 mini_kdc.cc:317] Time spent kinit for test-admin: real 0.124s user 0.005s sys 0.018s
Aug 07 11:30:22 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:22 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:30:22.809268 1872 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:22.776439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.24.103.126:44615)
0807 11:30:22.795522 (+ 19083us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:22.795528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:22.795555 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:30:22.795621 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:22.795625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:22.795708 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:22.795876 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:22.797880 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.799205 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.801300 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.802042 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.802139 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.808932 (+ 6793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39052: BlockingRecv error: recv got EOF from 127.0.0.1:39052 (error 108)
Metrics: {"server-negotiator.queue_time_us":18936,"thread_start_us":18819,"threads_started":1}
Aug 07 11:30:22 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM
I20260807 11:30:22.898176 1872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:22.851110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.24.103.126:44615)
0807 11:30:22.851329 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:22.851336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:22.869478 (+ 18142us) server_negotiation.cc:407] Connection header received
0807 11:30:22.875506 (+ 6028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:22.875514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:22.875632 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:22.875721 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:22.878074 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.879428 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.881564 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:22.882296 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:22.882411 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:22.890614 (+ 8203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:22.890622 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:22.890627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:22.890690 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:22.894515 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:22.895548 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:22.895555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:22.895559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:22.895738 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:22.896290 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:22.896296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:22.896300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:22.896671 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:22.896935 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:22.897535 (+ 600us) server_negotiation.cc:299] Negotiation successful
0807 11:30:22.897799 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:30:22.903318 1226 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:30:22.904131 1226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:30:22.914136 1379 tablet_service.cc:1467] Processing CreateTablet for tablet 00f26fd44ffd4a878e85879180752e64 (DEFAULT_TABLE table=test-table [id=3145a947f6284cc0a401cbebdb9d86f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:22.914644 1379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f26fd44ffd4a878e85879180752e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:22.919500 1536 tablet_service.cc:1467] Processing CreateTablet for tablet 00f26fd44ffd4a878e85879180752e64 (DEFAULT_TABLE table=test-table [id=3145a947f6284cc0a401cbebdb9d86f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:22.919984 1536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f26fd44ffd4a878e85879180752e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:22.923556 1688 tablet_service.cc:1467] Processing CreateTablet for tablet 00f26fd44ffd4a878e85879180752e64 (DEFAULT_TABLE table=test-table [id=3145a947f6284cc0a401cbebdb9d86f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:22.924161 1688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f26fd44ffd4a878e85879180752e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:22.939661 1878 tablet_bootstrap.cc:492] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18: Bootstrap starting.
I20260807 11:30:22.943257 1878 tablet_bootstrap.cc:654] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:22.944286 1881 tablet_bootstrap.cc:492] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232: Bootstrap starting.
I20260807 11:30:22.946993 1881 tablet_bootstrap.cc:654] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:22.948634 1880 tablet_bootstrap.cc:492] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc: Bootstrap starting.
I20260807 11:30:22.953712 1878 tablet_bootstrap.cc:492] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18: No bootstrap required, opened a new log
I20260807 11:30:22.953886 1878 ts_tablet_manager.cc:1397] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20260807 11:30:22.954705 1878 raft_consensus.cc:348] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.954999 1878 raft_consensus.cc:374] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:22.955117 1878 raft_consensus.cc:729] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f47045eb9d47a2a8da13f1f5598e18, State: Initialized, Role: FOLLOWER
I20260807 11:30:22.955403 1878 consensus_queue.cc:260] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.959985 1878 ts_tablet_manager.cc:1428] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260807 11:30:22.962383 1880 tablet_bootstrap.cc:654] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:22.965293 1881 tablet_bootstrap.cc:492] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232: No bootstrap required, opened a new log
I20260807 11:30:22.965483 1881 ts_tablet_manager.cc:1397] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.013s
I20260807 11:30:22.967728 1881 raft_consensus.cc:348] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.968024 1881 raft_consensus.cc:374] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:22.968133 1881 raft_consensus.cc:729] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5688689f848d4c818b3a1a0fdb6ec232, State: Initialized, Role: FOLLOWER
I20260807 11:30:22.968456 1881 consensus_queue.cc:260] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.969076 1881 ts_tablet_manager.cc:1428] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 11:30:22.973652 1880 tablet_bootstrap.cc:492] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc: No bootstrap required, opened a new log
I20260807 11:30:22.973865 1880 ts_tablet_manager.cc:1397] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20260807 11:30:22.976540 1880 raft_consensus.cc:348] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.976850 1880 raft_consensus.cc:374] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:22.976967 1880 raft_consensus.cc:729] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcea41ec2e954d4faeccf1bcabd3b4cc, State: Initialized, Role: FOLLOWER
I20260807 11:30:22.977243 1880 consensus_queue.cc:260] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:22.979544 1880 ts_tablet_manager.cc:1428] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260807 11:30:23.214540 1829 raft_consensus.cc:482] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:30:23.214834 1829 raft_consensus.cc:504] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.215775 1829 leader_election.cc:288] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989), 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67:43319)
I20260807 11:30:23.216399 1561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f26fd44ffd4a878e85879180752e64" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f47045eb9d47a2a8da13f1f5598e18" is_pre_election: true
I20260807 11:30:23.216568 1715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f26fd44ffd4a878e85879180752e64" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5688689f848d4c818b3a1a0fdb6ec232" is_pre_election: true
I20260807 11:30:23.216708 1561 raft_consensus.cc:2454] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 0.
I20260807 11:30:23.216863 1715 raft_consensus.cc:2454] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 0.
I20260807 11:30:23.217444 1326 leader_election.cc:302] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5688689f848d4c818b3a1a0fdb6ec232, bcea41ec2e954d4faeccf1bcabd3b4cc; no voters:
I20260807 11:30:23.217800 1829 raft_consensus.cc:2781] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:30:23.217927 1829 raft_consensus.cc:482] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:30:23.218065 1829 raft_consensus.cc:3037] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:23.220605 1829 raft_consensus.cc:504] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.221297 1829 leader_election.cc:288] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 election: Requested vote from peers 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989), 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67:43319)
I20260807 11:30:23.222002 1561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f26fd44ffd4a878e85879180752e64" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f47045eb9d47a2a8da13f1f5598e18"
I20260807 11:30:23.222280 1561 raft_consensus.cc:3037] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:23.222277 1715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f26fd44ffd4a878e85879180752e64" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5688689f848d4c818b3a1a0fdb6ec232"
I20260807 11:30:23.222530 1715 raft_consensus.cc:3037] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:23.225915 1561 raft_consensus.cc:2454] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 1.
I20260807 11:30:23.226493 1325 leader_election.cc:302] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f47045eb9d47a2a8da13f1f5598e18, bcea41ec2e954d4faeccf1bcabd3b4cc; no voters:
I20260807 11:30:23.226795 1715 raft_consensus.cc:2454] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 1.
I20260807 11:30:23.227061 1829 raft_consensus.cc:2781] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:23.227873 1829 raft_consensus.cc:686] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 LEADER]: Becoming Leader. State: Replica: bcea41ec2e954d4faeccf1bcabd3b4cc, State: Running, Role: LEADER
I20260807 11:30:23.228266 1829 consensus_queue.cc:237] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.231698 1228 catalog_manager.cc:5495] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc reported cstate change: term changed from 0 to 1, leader changed from <none> to bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65). New cstate: current_term: 1 leader_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } health_report { overall_health: UNKNOWN } } }
I20260807 11:30:23.319242 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:23.312005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.24.103.65:35005)
0807 11:30:23.312625 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:23.312630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:23.312646 (+ 16us) server_negotiation.cc:407] Connection header received
0807 11:30:23.312705 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:23.312709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:23.312777 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:23.312860 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:30:23.313458 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:23.314526 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:23.316910 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:23.317403 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:23.317509 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:23.318155 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:30:23.318295 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:30:23.318439 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:23.318629 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 11:30:23.318723 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":286,"threads_started":1}
I20260807 11:30:23.322944 1715 raft_consensus.cc:1261] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Refusing update from remote peer bcea41ec2e954d4faeccf1bcabd3b4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:23.323148 1561 raft_consensus.cc:1261] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Refusing update from remote peer bcea41ec2e954d4faeccf1bcabd3b4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:23.323846 1829 consensus_queue.cc:1035] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:23.324218 1885 consensus_queue.cc:1035] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:23.348879 1226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:23.349205 1226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:23.354240 1226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949: Sending DeleteTablet for 3 replicas of tablet 00f26fd44ffd4a878e85879180752e64
I20260807 11:30:23.355691 1688 tablet_service.cc:1514] Processing DeleteTablet for tablet 00f26fd44ffd4a878e85879180752e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:23 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:51020
I20260807 11:30:23.356598 1536 tablet_service.cc:1514] Processing DeleteTablet for tablet 00f26fd44ffd4a878e85879180752e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:23 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:57244
I20260807 11:30:23.356607 1379 tablet_service.cc:1514] Processing DeleteTablet for tablet 00f26fd44ffd4a878e85879180752e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:23 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:39876
I20260807 11:30:23.357721 1228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:30:23.358700 1228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:30:23.370013 1689 tablet_service.cc:1467] Processing CreateTablet for tablet a87dda848e864c5c8e8619833ca8d2b5 (DEFAULT_TABLE table=test-table [id=99091d3c63934482894e26ebd30531b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:23.370321 1378 tablet_service.cc:1467] Processing CreateTablet for tablet a87dda848e864c5c8e8619833ca8d2b5 (DEFAULT_TABLE table=test-table [id=99091d3c63934482894e26ebd30531b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:23.370669 1689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87dda848e864c5c8e8619833ca8d2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:23.370913 1378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87dda848e864c5c8e8619833ca8d2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:23.375888 1535 tablet_service.cc:1467] Processing CreateTablet for tablet a87dda848e864c5c8e8619833ca8d2b5 (DEFAULT_TABLE table=test-table [id=99091d3c63934482894e26ebd30531b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:23.376525 1535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87dda848e864c5c8e8619833ca8d2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:23.379822 1900 tablet_replica.cc:330] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc: stopping tablet replica
I20260807 11:30:23.380255 1900 raft_consensus.cc:2229] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:30:23.381194 1900 raft_consensus.cc:2258] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:23.383900 1899 tablet_replica.cc:330] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18: stopping tablet replica
I20260807 11:30:23.384126 1899 raft_consensus.cc:2229] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:23.384400 1899 raft_consensus.cc:2258] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:23.386822 1881 tablet_bootstrap.cc:492] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232: Bootstrap starting.
I20260807 11:30:23.388912 1881 tablet_bootstrap.cc:654] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:23.390076 1880 tablet_bootstrap.cc:492] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc: Bootstrap starting.
I20260807 11:30:23.392097 1880 tablet_bootstrap.cc:654] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:23.392674 1878 tablet_bootstrap.cc:492] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18: Bootstrap starting.
I20260807 11:30:23.393560 1900 ts_tablet_manager.cc:1905] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:23.394848 1878 tablet_bootstrap.cc:654] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:23.396920 1899 ts_tablet_manager.cc:1905] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:23.397428 1898 tablet_replica.cc:330] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232: stopping tablet replica
I20260807 11:30:23.397716 1898 raft_consensus.cc:2229] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:23.398074 1898 raft_consensus.cc:2258] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:23.399669 1898 ts_tablet_manager.cc:1905] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:23.402215 1900 ts_tablet_manager.cc:1918] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:23.402385 1900 log.cc:1196] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/00f26fd44ffd4a878e85879180752e64
I20260807 11:30:23.402866 1900 ts_tablet_manager.cc:1939] T 00f26fd44ffd4a878e85879180752e64 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting consensus metadata
I20260807 11:30:23.404080 1211 catalog_manager.cc:4841] TS bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65:35005): tablet 00f26fd44ffd4a878e85879180752e64 (table test-table [id=3145a947f6284cc0a401cbebdb9d86f1]) successfully deleted
I20260807 11:30:23.406076 1898 ts_tablet_manager.cc:1918] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:23.406255 1898 log.cc:1196] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/00f26fd44ffd4a878e85879180752e64
I20260807 11:30:23.406695 1898 ts_tablet_manager.cc:1939] T 00f26fd44ffd4a878e85879180752e64 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting consensus metadata
I20260807 11:30:23.407876 1212 catalog_manager.cc:4841] TS 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67:43319): tablet 00f26fd44ffd4a878e85879180752e64 (table test-table [id=3145a947f6284cc0a401cbebdb9d86f1]) successfully deleted
I20260807 11:30:23.408391 1899 ts_tablet_manager.cc:1918] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:23.408643 1899 log.cc:1196] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/00f26fd44ffd4a878e85879180752e64
I20260807 11:30:23.409255 1899 ts_tablet_manager.cc:1939] T 00f26fd44ffd4a878e85879180752e64 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting consensus metadata
I20260807 11:30:23.410734 1211 catalog_manager.cc:4841] TS 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989): tablet 00f26fd44ffd4a878e85879180752e64 (table test-table [id=3145a947f6284cc0a401cbebdb9d86f1]) successfully deleted
I20260807 11:30:23.412197 1878 tablet_bootstrap.cc:492] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18: No bootstrap required, opened a new log
I20260807 11:30:23.412400 1878 ts_tablet_manager.cc:1397] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.011s
I20260807 11:30:23.413111 1880 tablet_bootstrap.cc:492] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc: No bootstrap required, opened a new log
I20260807 11:30:23.413329 1880 ts_tablet_manager.cc:1397] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.003s
I20260807 11:30:23.413378 1878 raft_consensus.cc:348] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.413671 1878 raft_consensus.cc:374] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:23.413790 1881 tablet_bootstrap.cc:492] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232: No bootstrap required, opened a new log
I20260807 11:30:23.413775 1878 raft_consensus.cc:729] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f47045eb9d47a2a8da13f1f5598e18, State: Initialized, Role: FOLLOWER
I20260807 11:30:23.413968 1881 ts_tablet_manager.cc:1397] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.008s
I20260807 11:30:23.414069 1878 consensus_queue.cc:260] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.414184 1880 raft_consensus.cc:348] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.414453 1880 raft_consensus.cc:374] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:23.414559 1880 raft_consensus.cc:729] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcea41ec2e954d4faeccf1bcabd3b4cc, State: Initialized, Role: FOLLOWER
I20260807 11:30:23.414645 1878 ts_tablet_manager.cc:1428] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260807 11:30:23.414810 1880 consensus_queue.cc:260] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.415319 1880 ts_tablet_manager.cc:1428] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260807 11:30:23.415465 1881 raft_consensus.cc:348] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.415750 1881 raft_consensus.cc:374] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:23.415861 1881 raft_consensus.cc:729] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5688689f848d4c818b3a1a0fdb6ec232, State: Initialized, Role: FOLLOWER
I20260807 11:30:23.416106 1881 consensus_queue.cc:260] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.417263 1881 ts_tablet_manager.cc:1428] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 11:30:23.657935 1829 raft_consensus.cc:482] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:30:23.658222 1829 raft_consensus.cc:504] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.658982 1829 leader_election.cc:288] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989), 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67:43319)
I20260807 11:30:23.659617 1561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87dda848e864c5c8e8619833ca8d2b5" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f47045eb9d47a2a8da13f1f5598e18" is_pre_election: true
I20260807 11:30:23.659651 1715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87dda848e864c5c8e8619833ca8d2b5" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5688689f848d4c818b3a1a0fdb6ec232" is_pre_election: true
I20260807 11:30:23.659870 1561 raft_consensus.cc:2454] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 0.
I20260807 11:30:23.659878 1715 raft_consensus.cc:2454] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 0.
I20260807 11:30:23.660352 1326 leader_election.cc:302] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5688689f848d4c818b3a1a0fdb6ec232, bcea41ec2e954d4faeccf1bcabd3b4cc; no voters:
I20260807 11:30:23.660666 1829 raft_consensus.cc:2781] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:30:23.660780 1829 raft_consensus.cc:482] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:30:23.660917 1829 raft_consensus.cc:3037] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:23.663043 1829 raft_consensus.cc:504] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.663663 1829 leader_election.cc:288] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 election: Requested vote from peers 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989), 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67:43319)
I20260807 11:30:23.664198 1715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87dda848e864c5c8e8619833ca8d2b5" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5688689f848d4c818b3a1a0fdb6ec232"
I20260807 11:30:23.664245 1561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87dda848e864c5c8e8619833ca8d2b5" candidate_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f47045eb9d47a2a8da13f1f5598e18"
I20260807 11:30:23.664446 1561 raft_consensus.cc:3037] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:23.664449 1715 raft_consensus.cc:3037] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:23.667480 1561 raft_consensus.cc:2454] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 1.
I20260807 11:30:23.667480 1715 raft_consensus.cc:2454] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcea41ec2e954d4faeccf1bcabd3b4cc in term 1.
I20260807 11:30:23.667925 1325 leader_election.cc:302] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f47045eb9d47a2a8da13f1f5598e18, bcea41ec2e954d4faeccf1bcabd3b4cc; no voters:
I20260807 11:30:23.668231 1829 raft_consensus.cc:2781] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:23.668450 1829 raft_consensus.cc:686] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 LEADER]: Becoming Leader. State: Replica: bcea41ec2e954d4faeccf1bcabd3b4cc, State: Running, Role: LEADER
I20260807 11:30:23.668720 1829 consensus_queue.cc:237] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } }
I20260807 11:30:23.671581 1226 catalog_manager.cc:5495] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc reported cstate change: term changed from 0 to 1, leader changed from <none> to bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65). New cstate: current_term: 1 leader_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea41ec2e954d4faeccf1bcabd3b4cc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 35005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 } health_report { overall_health: UNKNOWN } } }
I20260807 11:30:23.721957 1561 raft_consensus.cc:1261] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Refusing update from remote peer bcea41ec2e954d4faeccf1bcabd3b4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:23.722021 1715 raft_consensus.cc:1261] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Refusing update from remote peer bcea41ec2e954d4faeccf1bcabd3b4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:23.722656 1829 consensus_queue.cc:1035] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5688689f848d4c818b3a1a0fdb6ec232" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:23.722958 1885 consensus_queue.cc:1035] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f47045eb9d47a2a8da13f1f5598e18" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 40989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:23.776239 1226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:23.776486 1226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:23.779562 1226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3044f1c76144a75a313adab9d4c3949: Sending DeleteTablet for 3 replicas of tablet a87dda848e864c5c8e8619833ca8d2b5
I20260807 11:30:23.780483 1378 tablet_service.cc:1514] Processing DeleteTablet for tablet a87dda848e864c5c8e8619833ca8d2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:23 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:39876
I20260807 11:30:23.780611 1535 tablet_service.cc:1514] Processing DeleteTablet for tablet a87dda848e864c5c8e8619833ca8d2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:23 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:57244
I20260807 11:30:23.780643 1689 tablet_service.cc:1514] Processing DeleteTablet for tablet a87dda848e864c5c8e8619833ca8d2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:23 UTC) from {username='oryx', principal='oryx/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:51020
I20260807 11:30:23.780946 1899 tablet_replica.cc:330] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18: stopping tablet replica
I20260807 11:30:23.780957 1900 tablet_replica.cc:330] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc: stopping tablet replica
I20260807 11:30:23.781189 1899 raft_consensus.cc:2229] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:23.781189 1900 raft_consensus.cc:2229] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:30:23.781390 1899 raft_consensus.cc:2258] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:23.781483 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 1309
I20260807 11:30:23.781539 1900 raft_consensus.cc:2258] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:23.781980 1898 tablet_replica.cc:330] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232: stopping tablet replica
I20260807 11:30:23.782261 1898 raft_consensus.cc:2229] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:23.782496 1898 raft_consensus.cc:2258] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:23.784212 1898 ts_tablet_manager.cc:1905] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:23.784193 1899 ts_tablet_manager.cc:1905] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:23.784310 1900 ts_tablet_manager.cc:1905] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:23.791364 1898 ts_tablet_manager.cc:1918] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:30:23.791411 1900 ts_tablet_manager.cc:1918] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:30:23.791554 1898 log.cc:1196] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/a87dda848e864c5c8e8619833ca8d2b5
I20260807 11:30:23.791576 1900 log.cc:1196] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/a87dda848e864c5c8e8619833ca8d2b5
I20260807 11:30:23.791682 1899 ts_tablet_manager.cc:1918] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 11:30:23.791877 1899 log.cc:1196] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/a87dda848e864c5c8e8619833ca8d2b5
I20260807 11:30:23.792060 1900 ts_tablet_manager.cc:1939] T a87dda848e864c5c8e8619833ca8d2b5 P bcea41ec2e954d4faeccf1bcabd3b4cc: Deleting consensus metadata
I20260807 11:30:23.792062 1898 ts_tablet_manager.cc:1939] T a87dda848e864c5c8e8619833ca8d2b5 P 5688689f848d4c818b3a1a0fdb6ec232: Deleting consensus metadata
I20260807 11:30:23.792402 1899 ts_tablet_manager.cc:1939] T a87dda848e864c5c8e8619833ca8d2b5 P 92f47045eb9d47a2a8da13f1f5598e18: Deleting consensus metadata
I20260807 11:30:23.793227 1212 catalog_manager.cc:4841] TS 5688689f848d4c818b3a1a0fdb6ec232 (127.24.103.67:43319): tablet a87dda848e864c5c8e8619833ca8d2b5 (table test-table [id=99091d3c63934482894e26ebd30531b8]) successfully deleted
I20260807 11:30:23.793763 1211 catalog_manager.cc:4841] TS bcea41ec2e954d4faeccf1bcabd3b4cc (127.24.103.65:35005): tablet a87dda848e864c5c8e8619833ca8d2b5 (table test-table [id=99091d3c63934482894e26ebd30531b8]) successfully deleted
I20260807 11:30:23.794030 1211 catalog_manager.cc:4841] TS 92f47045eb9d47a2a8da13f1f5598e18 (127.24.103.66:40989): tablet a87dda848e864c5c8e8619833ca8d2b5 (table test-table [id=99091d3c63934482894e26ebd30531b8]) successfully deleted
Aug 07 11:30:23 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.65@KRBTEST.COM
I20260807 11:30:23.801857 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:23.782432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.24.103.65:35005)
0807 11:30:23.782608 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:23.782614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:23.783244 (+ 630us) server_negotiation.cc:407] Connection header received
0807 11:30:23.783537 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:23.783543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:23.783633 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:23.783711 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:23.785772 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:23.786945 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:23.788828 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:23.789559 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:23.789696 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:23.795757 (+ 6061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:23.795782 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:23.795799 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:23.795839 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:23.798781 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:23.799642 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:23.799652 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:23.799655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:23.799810 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:23.800222 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:23.800227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:23.800232 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:23.800607 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:23.800946 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:23.801270 (+ 324us) server_negotiation.cc:299] Negotiation successful
0807 11:30:23.801497 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 11:30:23.802503 1455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:23.932669 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 1309
I20260807 11:30:23.960706 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 1464
Aug 07 11:30:23 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.66@KRBTEST.COM
I20260807 11:30:23.977598 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:23.961568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.24.103.66:40989)
0807 11:30:23.962058 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:23.962063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:23.962079 (+ 16us) server_negotiation.cc:407] Connection header received
0807 11:30:23.962135 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:23.962139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:23.962200 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:23.962323 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:23.963714 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:23.964490 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:23.965766 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:23.966198 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:23.966335 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:23.970963 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:23.970980 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:23.970996 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:23.971095 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:23.974393 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:23.975284 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:23.975292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:23.975295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:23.975440 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:23.975869 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:23.975873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:23.975876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:23.976283 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:23.976646 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:23.976976 (+ 330us) server_negotiation.cc:299] Negotiation successful
0807 11:30:23.977227 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":247,"threads_started":1}
I20260807 11:30:23.978423 1610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:24.082064 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 1464
I20260807 11:30:24.104669 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 1619
Aug 07 11:30:24 dist-test-slave-wl2h krb5kdc[1180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.103.67@KRBTEST.COM
I20260807 11:30:24.120951 1923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:24.105508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.24.103.67:43319)
0807 11:30:24.106020 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:24.106025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:24.106042 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:30:24.106099 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:24.106103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:24.106165 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:24.106286 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:24.107739 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:24.108560 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:24.109789 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:24.110181 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:24.110280 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:24.115289 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:24.115304 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:24.115321 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:24.115364 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:24.118263 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:24.119064 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:24.119076 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:24.119078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:24.119214 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:24.119615 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:24.119619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:24.119621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:24.119960 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:24.120190 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:24.120507 (+ 317us) server_negotiation.cc:299] Negotiation successful
0807 11:30:24.120671 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":306,"threads_started":1}
I20260807 11:30:24.121599 1763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:24.270001 1926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:24.268857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:35005 (local address 127.24.103.67:49705)
0807 11:30:24.269659 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:24.269810 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":580,"threads_started":1}
I20260807 11:30:24.270081 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 1619
I20260807 11:30:24.298125 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 1193
I20260807 11:30:24.299491 1275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:24.423098 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 1193
2026-08-07T11:30:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6853 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:30:24 dist-test-slave-wl2h krb5kdc[1930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:30:24 dist-test-slave-wl2h krb5kdc[1930](info): set up 2 sockets
Aug 07 11:30:24 dist-test-slave-wl2h krb5kdc[1930](info): commencing operation
krb5kdc: starting...
W20260807 11:30:25.662557 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.005s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:30:25 dist-test-slave-wl2h krb5kdc[1930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:30:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:30:25.972291 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:46003
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:46003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:26.320389 1944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:26.320760 1944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:26.320919 1944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:26.328234 1944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:26.328392 1944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:26.328504 1944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:26.328609 1944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:26.328691 1944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:30:26.338790 1944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:46003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:46003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:26.340225 1944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:26.341569 1944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:26.353957 1953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:26.353976 1956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:26.354151 1954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:26.354722 1944 server_base.cc:1034] running on GCE node
I20260807 11:30:26.355618 1944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:26.357182 1944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:26.358440 1944 hybrid_clock.cc:648] HybridClock initialized: now 1786102226358344 us; error 78 us; skew 500 ppm
I20260807 11:30:26.358825 1944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T11:30:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2011 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:30:26 dist-test-slave-wl2h krb5kdc[1964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:30:26 dist-test-slave-wl2h krb5kdc[1964](info): set up 2 sockets
Aug 07 11:30:26 dist-test-slave-wl2h krb5kdc[1964](info): commencing operation
krb5kdc: starting...
W20260807 11:30:27.665099 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.002s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:30:27 dist-test-slave-wl2h krb5kdc[1964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:30:27.964170 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:32981
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:28.313439 1977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:28.313766 1977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:28.313949 1977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:28.321591 1977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:28.321712 1977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:28.321848 1977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:28.321985 1977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:28.322070 1977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:30:28.332470 1977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:32981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:28.334093 1977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:28.335567 1977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:28.349294 1982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:28.349318 1983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:28.349277 1985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:28.350435 1977 server_base.cc:1034] running on GCE node
I20260807 11:30:28.351362 1977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:28.352862 1977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:28.354103 1977 hybrid_clock.cc:648] HybridClock initialized: now 1786102228354067 us; error 37 us; skew 500 ppm
I20260807 11:30:28.354504 1977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:28 dist-test-slave-wl2h krb5kdc[1964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102228, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:28.357945 1977 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:30:28.360406 1977 webserver.cc:460] Webserver started at http://127.24.103.126:42693/ using document root <none> and password file <none>
I20260807 11:30:28.361191 1977 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:28.361351 1977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:28.361685 1977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:28.364945 1977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "342d0cf68e9f49c5ac0641a25f8c807e"
format_stamp: "Formatted at 2026-08-07 11:30:28 on dist-test-slave-wl2h"
server_key: "985c09a400e8e88530346be6eb0b4c0a"
server_key_iv: "b2037736b1a35f6e3a77364b2bb9f2d9"
server_key_version: "encryptionkey@0"
I20260807 11:30:28.365765 1977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "342d0cf68e9f49c5ac0641a25f8c807e"
format_stamp: "Formatted at 2026-08-07 11:30:28 on dist-test-slave-wl2h"
server_key: "985c09a400e8e88530346be6eb0b4c0a"
server_key_iv: "b2037736b1a35f6e3a77364b2bb9f2d9"
server_key_version: "encryptionkey@0"
I20260807 11:30:28.371408 1977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:30:28.375937 1992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:28.377568 1977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260807 11:30:28.377774 1977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "342d0cf68e9f49c5ac0641a25f8c807e"
format_stamp: "Formatted at 2026-08-07 11:30:28 on dist-test-slave-wl2h"
server_key: "985c09a400e8e88530346be6eb0b4c0a"
server_key_iv: "b2037736b1a35f6e3a77364b2bb9f2d9"
server_key_version: "encryptionkey@0"
I20260807 11:30:28.377996 1977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:28.399117 1977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:28.404363 1977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:28.404745 1977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:28.427589 1977 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:32981
I20260807 11:30:28.427709 2045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:32981 every 8 connection(s)
I20260807 11:30:28.429246 1977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:30:28.433375 2046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:28.434192 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 1977
I20260807 11:30:28.434454 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:30:28.435218 24989 external_mini_cluster.cc:1428] Setting key b276238e2ac2c2af1a1e41ccc1216620
I20260807 11:30:28.449080 2046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e: Bootstrap starting.
Aug 07 11:30:28 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:28.453003 2046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:28.454337 2046 log.cc:824] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:28.458415 2046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e: No bootstrap required, opened a new log
I20260807 11:30:28.463546 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:28.437600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.24.103.126:32981)
0807 11:30:28.438575 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:28.438592 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:28.438646 (+ 54us) server_negotiation.cc:407] Connection header received
0807 11:30:28.440679 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:28.440712 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:28.441295 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:28.441803 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:28.443707 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:28.445899 (+ 2192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:28.447340 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:28.448432 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:28.448664 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:28.453657 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:28.453668 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:28.453687 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:28.453743 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:28.458072 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:28.459083 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:28.459092 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:28.459100 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:28.459348 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:28.459804 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:28.459810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:28.459813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:28.460577 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:28.460905 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:28.461448 (+ 543us) server_negotiation.cc:299] Negotiation successful
0807 11:30:28.461957 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":422,"threads_started":1}
I20260807 11:30:28.468715 2046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } }
I20260807 11:30:28.469132 2046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:28.469206 2046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342d0cf68e9f49c5ac0641a25f8c807e, State: Initialized, Role: FOLLOWER
I20260807 11:30:28.469736 2046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } }
I20260807 11:30:28.469960 2046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:30:28.470058 2046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:30:28.470185 2046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:28.472716 2046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } }
I20260807 11:30:28.473286 2046 leader_election.cc:302] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 342d0cf68e9f49c5ac0641a25f8c807e; no voters:
I20260807 11:30:28.473963 2046 leader_election.cc:288] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:30:28.474154 2052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:28.475060 2052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 1 LEADER]: Becoming Leader. State: Replica: 342d0cf68e9f49c5ac0641a25f8c807e, State: Running, Role: LEADER
I20260807 11:30:28.475621 2052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } }
I20260807 11:30:28.476357 2046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:28.478838 2054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342d0cf68e9f49c5ac0641a25f8c807e. Latest consensus state: current_term: 1 leader_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } } }
I20260807 11:30:28.479385 2054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:28.478822 2053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } } }
I20260807 11:30:28.480252 2053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:28.483166 2063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:30:28.489293 2063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:30:28.500378 2063 catalog_manager.cc:1347] Generated new cluster ID: 8be70b2defdc4513be007be343deb379
I20260807 11:30:28.500532 2063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:30:28.526430 2063 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:30:28.528007 2063 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:30:28.538599 2063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e: Generated new TSK 0
I20260807 11:30:28.539566 2063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:30:28.624171 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:32981
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:29.006131 2078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:29.006484 2078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:29.006656 2078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:29.014175 2078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:29.014356 2078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:29.014539 2078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:30:29.024657 2078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:32981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:29.026110 2078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:29.027585 2078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:29.041435 2083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:29.041486 2084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:29.041368 2086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:29.042425 2078 server_base.cc:1034] running on GCE node
I20260807 11:30:29.043182 2078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:29.044211 2078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:29.045471 2078 hybrid_clock.cc:648] HybridClock initialized: now 1786102229045438 us; error 52 us; skew 500 ppm
I20260807 11:30:29.045832 2078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:29 dist-test-slave-wl2h krb5kdc[1964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102229, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:29.049398 2078 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:30:29.050941 2078 webserver.cc:460] Webserver started at http://127.24.103.65:41801/ using document root <none> and password file <none>
I20260807 11:30:29.051666 2078 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:29.051816 2078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:29.052160 2078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:29.055312 2078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "d0317ca6cbe8466fa4bc2b6040c27040"
format_stamp: "Formatted at 2026-08-07 11:30:29 on dist-test-slave-wl2h"
server_key: "aa00cefcc44898775f72ab8d216ba959"
server_key_iv: "0245161c44802c3da49cedf0522933a5"
server_key_version: "encryptionkey@0"
I20260807 11:30:29.056125 2078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "d0317ca6cbe8466fa4bc2b6040c27040"
format_stamp: "Formatted at 2026-08-07 11:30:29 on dist-test-slave-wl2h"
server_key: "aa00cefcc44898775f72ab8d216ba959"
server_key_iv: "0245161c44802c3da49cedf0522933a5"
server_key_version: "encryptionkey@0"
I20260807 11:30:29.061264 2078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:30:29.064826 2094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:29.066130 2078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:30:29.066314 2078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "d0317ca6cbe8466fa4bc2b6040c27040"
format_stamp: "Formatted at 2026-08-07 11:30:29 on dist-test-slave-wl2h"
server_key: "aa00cefcc44898775f72ab8d216ba959"
server_key_iv: "0245161c44802c3da49cedf0522933a5"
server_key_version: "encryptionkey@0"
I20260807 11:30:29.066524 2078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:29.083956 2078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:29.088526 2078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:29.088853 2078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:29.089865 2078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:29.091367 2078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:29.091480 2078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:29.091643 2078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:29.091720 2078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:29.127244 2078 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:42969
I20260807 11:30:29.127305 2214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:42969 every 8 connection(s)
I20260807 11:30:29.128558 2078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:30:29.131506 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 2078
I20260807 11:30:29.131752 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:30:29.132645 24989 external_mini_cluster.cc:1428] Setting key 802ae4d6ee62b25d755881a70b418373
Aug 07 11:30:29 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102229, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:29.150373 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:29.131088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:58407 (local address 127.24.103.126:32981)
0807 11:30:29.131624 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:29.131633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:29.132580 (+ 947us) server_negotiation.cc:407] Connection header received
0807 11:30:29.134697 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:29.134702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:29.134786 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:29.134912 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:29.136449 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:29.137613 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:29.139199 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:29.139603 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:29.139747 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:29.143575 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:29.143584 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:29.143593 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:29.143626 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:29.146987 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:29.147745 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:29.147753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:29.147760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:29.147908 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:29.148327 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:29.148335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:29.148346 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:29.148748 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:29.148903 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:29.149638 (+ 735us) server_negotiation.cc:299] Negotiation successful
0807 11:30:29.149892 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":270,"threads_started":1}
I20260807 11:30:29.150954 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:29.131610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:58407)
0807 11:30:29.132411 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:29.132454 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:29.134416 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:29.135202 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:29.135210 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:29.135731 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:29.136274 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:29.136291 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:29.137768 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:29.137774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:29.138988 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:29.139001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:29.139843 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:29.139853 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:29.140109 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:29.140830 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:29.140856 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:29.143350 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:29.147167 (+ 3817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:29.147174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:29.147186 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:29.147604 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:29.148017 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:29.148020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:29.148022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:29.148207 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:29.148884 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:29.148890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:29.149236 (+ 346us) client_negotiation.cc:769] Sending connection context
0807 11:30:29.149459 (+ 223us) client_negotiation.cc:240] Negotiation successful
0807 11:30:29.149834 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":348,"threads_started":1}
I20260807 11:30:29.152691 2215 heartbeater.cc:347] Connected to a master server at 127.24.103.126:32981
I20260807 11:30:29.153012 2215 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:29.153973 2215 heartbeater.cc:511] Master 127.24.103.126:32981 requested a full tablet report, sending...
I20260807 11:30:29.156286 2009 ts_manager.cc:194] Registered new tserver with Master: d0317ca6cbe8466fa4bc2b6040c27040 (127.24.103.65:42969)
I20260807 11:30:29.157820 2009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:58407
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:30:29.202759 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:32981
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:29.555730 2224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:29.556074 2224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:29.556242 2224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:29.564016 2224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:29.564177 2224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:29.564354 2224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:30:29.574761 2224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:32981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:29.576201 2224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:29.577823 2224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:29.592176 2231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:29.592235 2232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:29.592186 2234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:29.593501 2224 server_base.cc:1034] running on GCE node
I20260807 11:30:29.594172 2224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:29.595299 2224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:29.596544 2224 hybrid_clock.cc:648] HybridClock initialized: now 1786102229596498 us; error 40 us; skew 500 ppm
I20260807 11:30:29.596890 2224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:29 dist-test-slave-wl2h krb5kdc[1964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102229, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:29.600347 2224 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:30:29.601892 2224 webserver.cc:460] Webserver started at http://127.24.103.66:44023/ using document root <none> and password file <none>
I20260807 11:30:29.602555 2224 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:29.602658 2224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:29.602934 2224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:29.605962 2224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "82c54b33ae134eeca075dffa32d66891"
format_stamp: "Formatted at 2026-08-07 11:30:29 on dist-test-slave-wl2h"
server_key: "47ffe447873691dd72d1ec23af1ebeea"
server_key_iv: "33043ff8cc675fa2f90976ce22f90541"
server_key_version: "encryptionkey@0"
I20260807 11:30:29.606765 2224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "82c54b33ae134eeca075dffa32d66891"
format_stamp: "Formatted at 2026-08-07 11:30:29 on dist-test-slave-wl2h"
server_key: "47ffe447873691dd72d1ec23af1ebeea"
server_key_iv: "33043ff8cc675fa2f90976ce22f90541"
server_key_version: "encryptionkey@0"
I20260807 11:30:29.612257 2224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 11:30:29.615974 2241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:29.617534 2224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:30:29.617722 2224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "82c54b33ae134eeca075dffa32d66891"
format_stamp: "Formatted at 2026-08-07 11:30:29 on dist-test-slave-wl2h"
server_key: "47ffe447873691dd72d1ec23af1ebeea"
server_key_iv: "33043ff8cc675fa2f90976ce22f90541"
server_key_version: "encryptionkey@0"
I20260807 11:30:29.617928 2224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:29.639560 2224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:29.644343 2224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:29.644663 2224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:29.645798 2224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:29.647301 2224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:29.647385 2224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:29.647526 2224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:29.647578 2224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:29.684587 2224 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:44355
I20260807 11:30:29.684685 2369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:44355 every 8 connection(s)
I20260807 11:30:29.686017 2224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:30:29.692476 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 2224
I20260807 11:30:29.692727 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:30:29.693347 24989 external_mini_cluster.cc:1428] Setting key 6dd5ce6dad1cbbf758fbc609853494c0
Aug 07 11:30:29 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102229, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:29.710213 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:29.690243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:46221 (local address 127.24.103.126:32981)
0807 11:30:29.690785 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:29.690791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:29.690885 (+ 94us) server_negotiation.cc:407] Connection header received
0807 11:30:29.693433 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:29.693440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:29.693544 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:29.693741 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:29.695647 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:29.696483 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:29.699119 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:29.699602 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:29.699705 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:29.704214 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:29.704220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:29.704223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:29.704259 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:29.706951 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:29.707801 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:29.707806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:29.707811 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:29.707944 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:29.708412 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:29.708416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:29.708418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:29.708698 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:29.708857 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:29.709574 (+ 717us) server_negotiation.cc:299] Negotiation successful
0807 11:30:29.709827 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":259,"threads_started":1}
I20260807 11:30:29.711299 2372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:29.689656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:46221)
0807 11:30:29.690717 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:29.690766 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:29.693042 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:29.693976 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:29.693991 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:29.694728 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:29.695454 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:29.695483 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:29.696650 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:29.696658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:29.698894 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:29.698913 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:29.699857 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:29.699867 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:29.700212 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:29.701029 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:29.701071 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:29.703948 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:29.707178 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:29.707188 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:29.707213 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:29.707657 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:29.708065 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:29.708070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:29.708074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:29.708280 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:29.708833 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:29.708842 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:29.709200 (+ 358us) client_negotiation.cc:769] Sending connection context
0807 11:30:29.709468 (+ 268us) client_negotiation.cc:240] Negotiation successful
0807 11:30:29.710009 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":487,"threads_started":1}
I20260807 11:30:29.712832 2370 heartbeater.cc:347] Connected to a master server at 127.24.103.126:32981
I20260807 11:30:29.713176 2370 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:29.714118 2370 heartbeater.cc:511] Master 127.24.103.126:32981 requested a full tablet report, sending...
I20260807 11:30:29.715863 2009 ts_manager.cc:194] Registered new tserver with Master: 82c54b33ae134eeca075dffa32d66891 (127.24.103.66:44355)
I20260807 11:30:29.717173 2009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:46221
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:30:29.763896 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:32981
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:30.119623 2376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:30.119966 2376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:30.120136 2376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:30.127876 2376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:30.128029 2376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:30.128216 2376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:30:30.138177 2376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:32981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:30.139462 2376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:30.140784 2376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:30.154009 2383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:30.154239 2385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:30.154060 2382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:30.155315 2376 server_base.cc:1034] running on GCE node
I20260807 11:30:30.155911 2376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:30.156847 2376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:30.158085 2376 hybrid_clock.cc:648] HybridClock initialized: now 1786102230158045 us; error 52 us; skew 500 ppm
I20260807 11:30:30.158452 2376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:30:30.160969 2215 heartbeater.cc:503] Master 127.24.103.126:32981 was elected leader, sending a full tablet report...
Aug 07 11:30:30 dist-test-slave-wl2h krb5kdc[1964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:30.161866 2376 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:30:30.163527 2376 webserver.cc:460] Webserver started at http://127.24.103.67:41191/ using document root <none> and password file <none>
I20260807 11:30:30.164162 2376 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:30.164261 2376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:30.164513 2376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:30.167608 2376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "b8c679976732444881b5b70450f3d366"
format_stamp: "Formatted at 2026-08-07 11:30:30 on dist-test-slave-wl2h"
server_key: "bec2308b87d9b37899be7ae30c456b6c"
server_key_iv: "8fdc4308e2c3d9e778b1ba3fa58da749"
server_key_version: "encryptionkey@0"
I20260807 11:30:30.168377 2376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "b8c679976732444881b5b70450f3d366"
format_stamp: "Formatted at 2026-08-07 11:30:30 on dist-test-slave-wl2h"
server_key: "bec2308b87d9b37899be7ae30c456b6c"
server_key_iv: "8fdc4308e2c3d9e778b1ba3fa58da749"
server_key_version: "encryptionkey@0"
I20260807 11:30:30.173682 2376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:30:30.177268 2392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:30.178746 2376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:30:30.178876 2376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "b8c679976732444881b5b70450f3d366"
format_stamp: "Formatted at 2026-08-07 11:30:30 on dist-test-slave-wl2h"
server_key: "bec2308b87d9b37899be7ae30c456b6c"
server_key_iv: "8fdc4308e2c3d9e778b1ba3fa58da749"
server_key_version: "encryptionkey@0"
I20260807 11:30:30.179116 2376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:30.196841 2376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:30.200951 2376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:30.201196 2376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:30.202100 2376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:30.203519 2376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:30.203591 2376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:30.203676 2376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:30.203722 2376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:30.238308 2376 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:46589
I20260807 11:30:30.238368 2512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:46589 every 8 connection(s)
I20260807 11:30:30.239627 2376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:30:30.246620 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 2376
I20260807 11:30:30.246834 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:30:30.247463 24989 external_mini_cluster.cc:1428] Setting key 94e81aa1adf39952b39450c9266f4146
Aug 07 11:30:30 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:30.260426 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:30.242107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:51261 (local address 127.24.103.126:32981)
0807 11:30:30.242621 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:30.242626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:30.243417 (+ 791us) server_negotiation.cc:407] Connection header received
0807 11:30:30.245272 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:30.245278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:30.245383 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:30.245506 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:30.247395 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:30.248286 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:30.250307 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:30.250973 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:30.251136 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:30.254586 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:30.254591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:30.254594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:30.254627 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:30.257519 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:30.258230 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:30.258235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:30.258237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:30.258380 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:30.258790 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:30.258794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:30.258797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:30.259196 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:30.259354 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:30.259968 (+ 614us) server_negotiation.cc:299] Negotiation successful
0807 11:30:30.260136 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":273,"threads_started":1}
I20260807 11:30:30.261452 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:30.242504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:51261)
0807 11:30:30.243252 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:30.243303 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:30.244997 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:30.245742 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:30.245750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:30.246358 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:30.247204 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:30.247230 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:30.248460 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:30.248464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:30.250154 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:30.250166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:30.251210 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:30.251215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:30.251414 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:30.252136 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:30.252164 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:30.254383 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:30.257667 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:30.257674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:30.257686 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:30.258098 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:30.258484 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:30.258487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:30.258489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:30.258663 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:30.259332 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:30.259338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:30.259641 (+ 303us) client_negotiation.cc:769] Sending connection context
0807 11:30:30.259878 (+ 237us) client_negotiation.cc:240] Negotiation successful
0807 11:30:30.260263 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":288,"threads_started":1}
I20260807 11:30:30.263008 2513 heartbeater.cc:347] Connected to a master server at 127.24.103.126:32981
I20260807 11:30:30.263437 2513 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:30.264228 2513 heartbeater.cc:511] Master 127.24.103.126:32981 requested a full tablet report, sending...
I20260807 11:30:30.265713 2009 ts_manager.cc:194] Registered new tserver with Master: b8c679976732444881b5b70450f3d366 (127.24.103.67:46589)
I20260807 11:30:30.266486 2009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:51261
I20260807 11:30:30.275980 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
Entry for principal oryx/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab.
WARNING: no policy specified for oryx/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
Entry for principal oryx/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab.
WARNING: no policy specified for oryx/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
Entry for principal oryx/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab.
I20260807 11:30:30.466930 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 2078
Aug 07 11:30:30 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:30:30.486959 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:30.468168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.24.103.65:42969)
0807 11:30:30.468873 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:30.468885 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:30.469343 (+ 458us) server_negotiation.cc:407] Connection header received
0807 11:30:30.469564 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:30.469572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:30.469904 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:30.470076 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:30.471772 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:30.472849 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:30.474154 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:30.474738 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:30.474884 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:30.479829 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:30.479843 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:30.479859 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:30.479903 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:30.483609 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:30.484639 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:30.484646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:30.484649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:30.484795 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:30.485305 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:30.485309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:30.485311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:30.485726 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:30.485994 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:30.486440 (+ 446us) server_negotiation.cc:299] Negotiation successful
0807 11:30:30.486660 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":327,"threads_started":1}
I20260807 11:30:30.488380 2210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:30.573976 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 2078
I20260807 11:30:30.598824 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 2224
Aug 07 11:30:30 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:30:30.616501 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:30.599910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.24.103.66:44355)
0807 11:30:30.600657 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:30.600664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:30.600692 (+ 28us) server_negotiation.cc:407] Connection header received
0807 11:30:30.600755 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:30.600763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:30.600973 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:30.601116 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:30.602398 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:30.603621 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:30.604950 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:30.605516 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:30.605621 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:30.610299 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:30.610313 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:30.610328 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:30.610379 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:30.613393 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:30.614140 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:30.614147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:30.614149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:30.614282 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:30.614662 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:30.614666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:30.614668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:30.615224 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:30.615526 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:30.615839 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 11:30:30.616119 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":402,"threads_started":1}
I20260807 11:30:30.617852 2363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:30.704180 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 2224
I20260807 11:30:30.721146 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 2376
Aug 07 11:30:30 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:30:30.739274 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:30.722198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.24.103.67:46589)
0807 11:30:30.722796 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:30.722803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:30.722836 (+ 33us) server_negotiation.cc:407] Connection header received
0807 11:30:30.722899 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:30.722905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:30.723204 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:30.723344 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:30.724588 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:30.725583 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:30.727225 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:30.728208 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:30.728356 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:30.733374 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:30.733386 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:30.733401 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:30.733437 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:30.736244 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:30.736970 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:30.736980 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:30.736985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:30.737119 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:30.737514 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:30.737519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:30.737523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:30.737885 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:30.738202 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:30.738524 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 11:30:30.738794 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":271,"threads_started":1}
I20260807 11:30:30.740665 2507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:30.829902 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 2376
I20260807 11:30:30.850149 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 1977
I20260807 11:30:30.851606 2041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:30.958633 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 1977
I20260807 11:30:30.984342 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:32981
--webserver_interface=127.24.103.126
--webserver_port=42693
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:31.328414 2534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:31.328804 2534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:31.328967 2534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:31.336441 2534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:31.336601 2534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:31.336721 2534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:31.336834 2534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:31.336925 2534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:30:31.347105 2534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:32981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=42693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:31.348568 2534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:31.349994 2534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:31.362720 2539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:31.362780 2543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:31.362726 2540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:31.364017 2534 server_base.cc:1034] running on GCE node
I20260807 11:30:31.364921 2534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:31.366505 2534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:31.367748 2534 hybrid_clock.cc:648] HybridClock initialized: now 1786102231367704 us; error 70 us; skew 500 ppm
I20260807 11:30:31.368113 2534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:31 dist-test-slave-wl2h krb5kdc[1964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102231, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:31.378007 2534 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:30:31.382864 2534 webserver.cc:460] Webserver started at http://127.24.103.126:42693/ using document root <none> and password file <none>
I20260807 11:30:31.384191 2534 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:31.384392 2534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:31.391446 2534 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 11:30:31.394809 2551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:31.396543 2534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:30:31.396785 2534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "342d0cf68e9f49c5ac0641a25f8c807e"
format_stamp: "Formatted at 2026-08-07 11:30:28 on dist-test-slave-wl2h"
server_key: "985c09a400e8e88530346be6eb0b4c0a"
server_key_iv: "b2037736b1a35f6e3a77364b2bb9f2d9"
server_key_version: "encryptionkey@0"
I20260807 11:30:31.397917 2534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:31.413198 2534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:31.418046 2534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:31.418411 2534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:31.438979 2534 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:32981
I20260807 11:30:31.439175 2608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:32981 every 8 connection(s)
I20260807 11:30:31.440631 2534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:30:31.445104 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 2534
I20260807 11:30:31.446002 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:42969
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:32981
--webserver_port=41801
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.103.65
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:30:31.448125 2609 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:30:31.451630 2609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e: Bootstrap starting.
I20260807 11:30:31.477116 2609 log.cc:824] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:31.490325 2609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:30:31.491032 2609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e: Bootstrap complete.
I20260807 11:30:31.505126 2609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } }
I20260807 11:30:31.506345 2609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342d0cf68e9f49c5ac0641a25f8c807e, State: Initialized, Role: FOLLOWER
I20260807 11:30:31.507117 2609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } }
I20260807 11:30:31.507412 2609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:30:31.507543 2609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:30:31.507733 2609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:30:31.513670 2609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } }
I20260807 11:30:31.514326 2609 leader_election.cc:302] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 342d0cf68e9f49c5ac0641a25f8c807e; no voters:
I20260807 11:30:31.515070 2609 leader_election.cc:288] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 11:30:31.515292 2615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:30:31.516750 2615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [term 2 LEADER]: Becoming Leader. State: Replica: 342d0cf68e9f49c5ac0641a25f8c807e, State: Running, Role: LEADER
I20260807 11:30:31.517385 2615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } }
I20260807 11:30:31.517505 2609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:31.520135 2616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } } }
I20260807 11:30:31.520069 2617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342d0cf68e9f49c5ac0641a25f8c807e. Latest consensus state: current_term: 2 leader_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342d0cf68e9f49c5ac0641a25f8c807e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 32981 } } }
I20260807 11:30:31.520498 2616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:31.520495 2617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:31.531497 2621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:30:31.538237 2621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:30:31.539813 2621 catalog_manager.cc:1259] Loaded cluster ID: 8be70b2defdc4513be007be343deb379
I20260807 11:30:31.539984 2621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:30:31.544174 2621 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:30:31.545776 2621 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 342d0cf68e9f49c5ac0641a25f8c807e: Loaded TSK: 0
I20260807 11:30:31.547483 2621 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 11:30:31.835685 2613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:31.835983 2613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:31.836127 2613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:31.843797 2613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:31.843926 2613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:31.844105 2613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:30:31.854547 2613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.65.keytab
--principal=oryx/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:42969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=41801
--tserver_master_addrs=127.24.103.126:32981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:31.856117 2613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:31.857563 2613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:31.872802 2639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:31.872941 2638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:31.873539 2642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:31.874517 2613 server_base.cc:1034] running on GCE node
I20260807 11:30:31.875391 2613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:31.876647 2613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:31.877938 2613 hybrid_clock.cc:648] HybridClock initialized: now 1786102231877857 us; error 61 us; skew 500 ppm
I20260807 11:30:31.878326 2613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:31 dist-test-slave-wl2h krb5kdc[1964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102231, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:31.882468 2613 init.cc:377] Logged in from keytab as oryx/127.24.103.65@KRBTEST.COM (short username oryx)
I20260807 11:30:31.884503 2613 webserver.cc:460] Webserver started at http://127.24.103.65:41801/ using document root <none> and password file <none>
I20260807 11:30:31.885516 2613 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:31.885686 2613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:31.893501 2613 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:30:31.897440 2651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:31.899187 2613 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:30:31.899362 2613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "d0317ca6cbe8466fa4bc2b6040c27040"
format_stamp: "Formatted at 2026-08-07 11:30:29 on dist-test-slave-wl2h"
server_key: "aa00cefcc44898775f72ab8d216ba959"
server_key_iv: "0245161c44802c3da49cedf0522933a5"
server_key_version: "encryptionkey@0"
I20260807 11:30:31.900730 2613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:31.926873 2613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:31.931665 2613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:31.932003 2613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:31.933099 2613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:31.934556 2613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:31.934645 2613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:31.934826 2613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:31.934928 2613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:31.968284 2613 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:42969
I20260807 11:30:31.968384 2776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:42969 every 8 connection(s)
I20260807 11:30:31.969859 2613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:30:31.973760 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 2613
I20260807 11:30:31.974787 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:44355
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:32981
--webserver_port=44023
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.103.66
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:30:31 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:31 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:31.993382 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:31.973391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:47243)
0807 11:30:31.974273 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:31.974332 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:31.976272 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:31.978521 (+ 2249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:31.978535 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:31.979276 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:31.979982 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:31.980002 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:31.982229 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:31.982239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:31.984484 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:31.984504 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:31.986163 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:31.986174 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:31.986507 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:31.987447 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:31.987477 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:31.991962 (+ 4485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":368,"threads_started":1}
W20260807 11:30:31.993424 2780 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:31.973592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:47243 (local address 127.24.103.126:32981)
0807 11:30:31.977217 (+ 3625us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:31.977230 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:31.977272 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:30:31.977603 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:31.977616 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:31.977967 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:31.978374 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:31.980231 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:31.982006 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:31.984702 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:31.985845 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:31.987083 (+ 1238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:31.992357 (+ 5274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:47243: BlockingRecv error: recv got EOF from 127.24.103.65:47243 (error 108)
Metrics: {"server-negotiator.queue_time_us":3387,"thread_start_us":3172,"threads_started":1}
W20260807 11:30:31.994383 2777 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:32981 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.103.126:32981: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:32.005364 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:31.995191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:34759)
0807 11:30:31.995422 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:31.995444 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:31.995561 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:31.995910 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:31.995917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:31.996463 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:31.996685 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:31.996700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:31.997813 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:31.997821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:31.999589 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:31.999609 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.000398 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.000407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:32.000659 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:32.001467 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:32.001498 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:32.005110 (+ 3612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260807 11:30:32.005813 2780 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:31.995355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:34759 (local address 127.24.103.126:32981)
0807 11:30:31.995513 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:31.995517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:31.995536 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:30:31.995694 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:31.995697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:31.995763 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:31.995850 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:31.996852 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:31.997643 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:31.999753 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.000210 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.000321 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:32.005444 (+ 5123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:34759: BlockingRecv error: recv got EOF from 127.24.103.65:34759 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:32.015305 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:32.006284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:54985)
0807 11:30:32.006467 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:32.006489 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:32.006613 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:32.007092 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:32.007095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:32.007405 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:32.007600 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:32.007616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.009101 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.009105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:32.010140 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:32.010152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.011152 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.011157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:32.011327 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:32.011977 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:32.011995 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:32.014991 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 11:30:32.015558 2780 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:32.006345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:54985 (local address 127.24.103.126:32981)
0807 11:30:32.006530 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:32.006538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:32.006562 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:30:32.006760 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:32.006768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:32.006871 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:32.006988 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:32.007783 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.008943 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.010278 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.010961 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.011124 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:32.015282 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:54985: BlockingRecv error: recv got EOF from 127.24.103.65:54985 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260807 11:30:32.015719 2777 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 11:30:32.324517 2781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:32.324821 2781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:32.324962 2781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:32.332767 2781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:32.332898 2781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:32.333076 2781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:30:32.343596 2781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.66.keytab
--principal=oryx/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:44355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=44023
--tserver_master_addrs=127.24.103.126:32981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:32.345028 2781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:32.346705 2781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:32.360499 2788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:32.360654 2790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:32.360569 2787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:32.361670 2781 server_base.cc:1034] running on GCE node
I20260807 11:30:32.362433 2781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:32.363613 2781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:32.364938 2781 hybrid_clock.cc:648] HybridClock initialized: now 1786102232364879 us; error 43 us; skew 500 ppm
I20260807 11:30:32.365329 2781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102232, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:32.373586 2781 init.cc:377] Logged in from keytab as oryx/127.24.103.66@KRBTEST.COM (short username oryx)
I20260807 11:30:32.375253 2781 webserver.cc:460] Webserver started at http://127.24.103.66:44023/ using document root <none> and password file <none>
I20260807 11:30:32.376010 2781 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:32.376230 2781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:32.381968 2781 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 11:30:32.385185 2798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:32.386705 2781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:30:32.386901 2781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "82c54b33ae134eeca075dffa32d66891"
format_stamp: "Formatted at 2026-08-07 11:30:29 on dist-test-slave-wl2h"
server_key: "47ffe447873691dd72d1ec23af1ebeea"
server_key_iv: "33043ff8cc675fa2f90976ce22f90541"
server_key_version: "encryptionkey@0"
I20260807 11:30:32.388002 2781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:32.409541 2781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:32.413975 2781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:32.414270 2781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:32.415341 2781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:32.416813 2781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:32.416942 2781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:32.417078 2781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:32.417173 2781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:32.460266 2781 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:44355
I20260807 11:30:32.460362 2919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:44355 every 8 connection(s)
I20260807 11:30:32.461887 2781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:30:32.469143 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 2781
I20260807 11:30:32.469843 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:46589
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:32981
--webserver_port=41191
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.103.67
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:30:32.481891 2780 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:32.464963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:33455 (local address 127.24.103.126:32981)
0807 11:30:32.465234 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:32.465239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:32.466470 (+ 1231us) server_negotiation.cc:407] Connection header received
0807 11:30:32.468508 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:32.468544 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:32.468657 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:32.468815 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:32.471980 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.473266 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.475245 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.476119 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.476276 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:32.481556 (+ 5280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:33455: BlockingRecv error: recv got EOF from 127.24.103.66:33455 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
I20260807 11:30:32.482359 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:32.465419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:33455)
0807 11:30:32.466286 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:32.466326 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:32.468142 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:32.469434 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:32.469444 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:32.470369 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:32.471687 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:32.471731 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.473508 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.473513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:32.474933 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:32.474948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.476419 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.476425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:32.476698 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:32.477490 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:32.477519 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:32.481222 (+ 3703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":412,"threads_started":1}
W20260807 11:30:32.483208 2920 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:32981 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.103.126:32981: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:32.493993 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:32.484008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:59399)
0807 11:30:32.484237 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:32.484255 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:32.484353 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:32.484750 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:32.484754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:32.485312 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:32.485472 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:32.485482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.487440 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.487445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:32.488388 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:32.488400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.489602 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.489607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:32.489758 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:32.490415 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:32.490433 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:32.493780 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260807 11:30:32.494412 2780 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:32.484125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:59399 (local address 127.24.103.126:32981)
0807 11:30:32.484344 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:32.484352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:32.484374 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:30:32.484458 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:32.484466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:32.484563 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:32.484668 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:32.485825 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.487250 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.488588 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.489359 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.489533 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:32.494107 (+ 4574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:59399: BlockingRecv error: recv got EOF from 127.24.103.66:59399 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:32.503307 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:32.494914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:44147)
0807 11:30:32.495092 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:32.495115 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:32.495234 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:32.495614 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:32.495618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:32.495924 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:32.496067 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:32.496074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.497092 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.497097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:32.497996 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:32.498007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.498709 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.498716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:32.498906 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:32.499631 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:32.499651 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:32.502959 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260807 11:30:32.503613 2920 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 11:30:32.503428 2780 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:32.495119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:44147 (local address 127.24.103.126:32981)
0807 11:30:32.495338 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:32.495343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:32.495359 (+ 16us) server_negotiation.cc:407] Connection header received
0807 11:30:32.495422 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:32.495426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:32.495482 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:32.495589 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:32.496216 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.496951 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.498138 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.498548 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.498679 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:32.503198 (+ 4519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:44147: BlockingRecv error: recv got EOF from 127.24.103.66:44147 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 11:30:32.839077 2923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:32.839485 2923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:32.839653 2923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:32.847168 2923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:32.847344 2923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:32.847532 2923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:30:32.858227 2923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/krb5kdc/oryx_127.24.103.67.keytab
--principal=oryx/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:46589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=41191
--tserver_master_addrs=127.24.103.126:32981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:32.859798 2923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:32.861416 2923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:32.875164 2931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:32.875272 2930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:32.875460 2933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:32.875921 2923 server_base.cc:1034] running on GCE node
I20260807 11:30:32.876605 2923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:32.877676 2923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:32.878911 2923 hybrid_clock.cc:648] HybridClock initialized: now 1786102232878866 us; error 36 us; skew 500 ppm
I20260807 11:30:32.879272 2923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:32 dist-test-slave-wl2h krb5kdc[1964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102232, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:32.882483 2923 init.cc:377] Logged in from keytab as oryx/127.24.103.67@KRBTEST.COM (short username oryx)
I20260807 11:30:32.884377 2923 webserver.cc:460] Webserver started at http://127.24.103.67:41191/ using document root <none> and password file <none>
I20260807 11:30:32.885159 2923 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:32.885336 2923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:32.891220 2923 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:30:32.894594 2941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:32.896114 2923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:30:32.896319 2923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "b8c679976732444881b5b70450f3d366"
format_stamp: "Formatted at 2026-08-07 11:30:30 on dist-test-slave-wl2h"
server_key: "bec2308b87d9b37899be7ae30c456b6c"
server_key_iv: "8fdc4308e2c3d9e778b1ba3fa58da749"
server_key_version: "encryptionkey@0"
I20260807 11:30:32.897358 2923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:32.941254 2923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:32.945955 2923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:32.946276 2923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:32.947391 2923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:32.948804 2923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:32.948947 2923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:32.949095 2923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:32.949188 2923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:32.985792 2923 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:46589
I20260807 11:30:32.985882 3066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:46589 every 8 connection(s)
I20260807 11:30:32.987412 2923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:30:32.995860 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 2923
Aug 07 11:30:33 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:33 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:30:33.007727 2780 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:32.990067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:43775 (local address 127.24.103.126:32981)
0807 11:30:32.990295 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:32.990302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:32.991731 (+ 1429us) server_negotiation.cc:407] Connection header received
0807 11:30:32.993924 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:32.993930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:32.994019 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:32.994112 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:32.996174 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.997216 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.999689 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.000221 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.000387 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:33.007408 (+ 7021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:43775: BlockingRecv error: recv got EOF from 127.24.103.67:43775 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 11:30:33.008400 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:32.990628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:43775)
0807 11:30:32.991522 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:32.991588 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:32.993644 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:32.994385 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:32.994393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:32.994967 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:32.995915 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:32.995937 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:32.997478 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:32.997483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:32.999503 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:32.999530 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.000444 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.000449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:33.000642 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:33.001444 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:33.001477 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:33.007157 (+ 5680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":424,"threads_started":1}
W20260807 11:30:33.009076 3068 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:32981 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.103.126:32981: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
I20260807 11:30:33.015825 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:32.997148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.24.103.126:32981)
0807 11:30:32.997959 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:32.997964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:32.997981 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:30:32.998073 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:32.998077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:32.998137 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:32.998238 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:33.000221 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.001271 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.003302 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.003952 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.004118 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:33.007811 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:33.007826 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:33.007851 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:33.007901 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:33.011662 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:33.012586 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:33.012597 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:33.012606 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:33.012889 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:33.013320 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:33.013326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:33.013330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:33.014032 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:33.014388 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:33.014859 (+ 471us) server_negotiation.cc:299] Negotiation successful
0807 11:30:33.015272 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":562,"threads_started":1}
Aug 07 11:30:33 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:33 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:33.020715 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:33.009696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:39381)
0807 11:30:33.009858 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:33.009878 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:33.009966 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:33.010551 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:33.010556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:33.010900 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:33.011101 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:33.011110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.012435 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.012471 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:33.014141 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:33.014159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.015170 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.015178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:33.015376 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:33.016352 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:33.016376 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:33.020261 (+ 3885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 11:30:33.020776 2780 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:33.010115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:39381 (local address 127.24.103.126:32981)
0807 11:30:33.010295 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:33.010298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:33.010312 (+ 14us) server_negotiation.cc:407] Connection header received
0807 11:30:33.010355 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:33.010358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:33.010411 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:33.010521 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:33.011250 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.012239 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.014300 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.014972 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.015132 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:33.020591 (+ 5459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:39381: BlockingRecv error: recv got EOF from 127.24.103.67:39381 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 07 11:30:33 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:33 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:33.029682 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:33.021480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:46947)
0807 11:30:33.021652 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:33.021670 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:33.021781 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:33.022119 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:33.022124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:33.022463 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:33.022603 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:33.022611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.023700 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.023706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:33.024632 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:33.024644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.025386 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.025393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:33.025591 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:33.026168 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:33.026185 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:33.029372 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 11:30:33.029824 2780 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:33.021659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:46947 (local address 127.24.103.126:32981)
0807 11:30:33.021869 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:33.021875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:33.021897 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:30:33.021935 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:33.021938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:33.021990 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:33.022091 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:33.022746 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.023560 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.024799 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.025226 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.025361 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:33.029566 (+ 4205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:46947: BlockingRecv error: recv got EOF from 127.24.103.67:46947 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 11:30:33.030042 3068 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 11:30:33 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:33 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:33.036329 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:33.016356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:36561)
0807 11:30:33.027442 (+ 11086us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:33.027458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:33.027570 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:33.027940 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:33.027944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:33.028409 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:33.028675 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:33.028693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.030104 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.030110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:33.031308 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:33.031323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.032219 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.032228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:33.032405 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:33.032924 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:33.032942 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:33.035981 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":10970,"thread_start_us":10849,"threads_started":1}
W20260807 11:30:33.036549 3072 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:33.016419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:36561 (local address 127.24.103.126:32981)
0807 11:30:33.016762 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:33.016771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:33.027559 (+ 10788us) server_negotiation.cc:407] Connection header received
0807 11:30:33.027704 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:33.027710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:33.027797 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:33.027910 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:33.028863 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.029606 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.031475 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.031966 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.032162 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:33.036269 (+ 4107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:36561: BlockingRecv error: recv got EOF from 127.24.103.65:36561 (error 108)
Metrics: {"server-negotiator.queue_time_us":187}
Aug 07 11:30:33 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:33 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:33.513206 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:33.504231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:45897)
0807 11:30:33.504799 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:33.504816 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:33.504984 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:33.505416 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:33.505421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:33.506030 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:33.506232 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:33.506242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.507365 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.507370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:33.508367 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:33.508379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.509117 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.509123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:33.509302 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:33.509921 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:33.509939 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:33.512927 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":293,"threads_started":1}
W20260807 11:30:33.513572 3072 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:33.504324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:45897 (local address 127.24.103.126:32981)
0807 11:30:33.504623 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:33.504630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:33.504906 (+ 276us) server_negotiation.cc:407] Connection header received
0807 11:30:33.505112 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:33.505117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:33.505254 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:33.505382 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:33.506390 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.507212 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.508516 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:33.508920 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:33.509064 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:33.513242 (+ 4178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:45897: BlockingRecv error: recv got EOF from 127.24.103.66:45897 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 07 11:30:34 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:34 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:34.041091 3075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:34.031057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:52517)
0807 11:30:34.031617 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:34.031642 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:34.031768 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:34.032291 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:34.032295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:34.032867 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:34.033067 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:34.033075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.034331 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.034338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:34.035377 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:34.035389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.036155 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.036161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:34.036354 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:34.037033 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:34.037057 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:34.040776 (+ 3719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":288,"threads_started":1}
W20260807 11:30:34.041198 3076 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:34.031184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:52517 (local address 127.24.103.126:32981)
0807 11:30:34.031985 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:34.031989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:34.032007 (+ 18us) server_negotiation.cc:407] Connection header received
0807 11:30:34.032074 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:34.032078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:34.032136 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:34.032230 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:34.033234 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.034182 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.035529 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.035960 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.036092 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:34.040955 (+ 4863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:52517: BlockingRecv error: recv got EOF from 127.24.103.67:52517 (error 108)
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":584,"threads_started":1}
Aug 07 11:30:34 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:34 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:34.047199 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:34.037610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:36211)
0807 11:30:34.038477 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:34.038494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:34.038615 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:34.039007 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:34.039033 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:34.039556 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:34.039760 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:34.039769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.041374 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.041380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:34.042298 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:34.042310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.043087 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.043093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:34.043249 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:34.043855 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:34.043873 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:34.046793 (+ 2920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":628,"threads_started":1}
W20260807 11:30:34.047423 3079 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:34.037723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:36211 (local address 127.24.103.126:32981)
0807 11:30:34.038338 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:34.038345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:34.038579 (+ 234us) server_negotiation.cc:407] Connection header received
0807 11:30:34.038745 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:34.038753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:34.038836 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:34.038945 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:34.039935 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.041166 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.042436 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.042849 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.042946 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:34.047110 (+ 4164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:36211: BlockingRecv error: recv got EOF from 127.24.103.65:36211 (error 108)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":258,"threads_started":1}
Aug 07 11:30:34 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:34 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:34.523978 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:34.514328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:33283)
0807 11:30:34.514929 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:34.514946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:34.515090 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:34.515437 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:34.515440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:34.515858 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:34.516035 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:34.516044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.517478 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.517486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:34.518969 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:34.518982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.519772 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.519778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:34.519955 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:34.520725 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:34.520744 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:34.523726 (+ 2982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":344,"threads_started":1}
W20260807 11:30:34.524192 3079 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:34.514594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:33283 (local address 127.24.103.126:32981)
0807 11:30:34.514790 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:34.514794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:34.515032 (+ 238us) server_negotiation.cc:407] Connection header received
0807 11:30:34.515208 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:34.515213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:34.515278 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:34.515369 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:34.516204 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.517308 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.519159 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:34.519588 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:34.519739 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:34.523951 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:33283: BlockingRecv error: recv got EOF from 127.24.103.66:33283 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 11:30:35 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:35 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:35.052791 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:35.042299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:37451)
0807 11:30:35.043358 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:35.043384 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:35.043555 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:35.044010 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:35.044015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:35.044468 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:35.044723 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:35.044738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.045909 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.045913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:35.046847 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:35.046859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.047664 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.047669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:35.047823 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:35.048984 (+ 1161us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:35.049014 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:35.052437 (+ 3423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":826,"threads_started":1}
W20260807 11:30:35.053020 3083 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:35.042573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:37451 (local address 127.24.103.126:32981)
0807 11:30:35.043266 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:35.043272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:35.043490 (+ 218us) server_negotiation.cc:407] Connection header received
0807 11:30:35.043685 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:35.043689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:35.043795 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:35.043919 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:35.044879 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.045776 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.046987 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.047495 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.047635 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:35.052660 (+ 5025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:37451: BlockingRecv error: recv got EOF from 127.24.103.67:37451 (error 108)
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":274,"threads_started":1}
Aug 07 11:30:35 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:35 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:35.058101 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:35.048231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:59019)
0807 11:30:35.048745 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:35.048771 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:35.048868 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:35.049378 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:35.049382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:35.049721 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:35.049890 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:35.049899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.051305 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.051312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:35.052394 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:35.052408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.053434 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.053441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:35.053651 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:35.054334 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:35.054354 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:35.057848 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":293,"threads_started":1}
W20260807 11:30:35.058396 3085 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:35.048353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:59019 (local address 127.24.103.126:32981)
0807 11:30:35.049020 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:35.049027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:35.049046 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:30:35.049104 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:35.049111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:35.049193 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:35.049330 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:35.050057 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.051144 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.052554 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.053260 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.053415 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:35.058100 (+ 4685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:59019: BlockingRecv error: recv got EOF from 127.24.103.65:59019 (error 108)
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":377,"threads_started":1}
Aug 07 11:30:35 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:35 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:35.534991 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:35.525217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:48905)
0807 11:30:35.525832 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:35.525851 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:35.526047 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:35.526478 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:35.526482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:35.526923 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:35.527137 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:35.527147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.528329 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.528335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:35.529351 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:35.529364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.530179 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.530185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:35.530391 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:35.531180 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:35.531205 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:35.534712 (+ 3507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":338,"threads_started":1}
W20260807 11:30:35.535279 3085 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:35.525369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:48905 (local address 127.24.103.126:32981)
0807 11:30:35.525635 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:35.525643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:35.525963 (+ 320us) server_negotiation.cc:407] Connection header received
0807 11:30:35.526232 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:35.526237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:35.526312 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:35.526442 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:35.527315 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.528148 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.529518 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:35.529997 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:35.530147 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:35.534949 (+ 4802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:48905: BlockingRecv error: recv got EOF from 127.24.103.66:48905 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 07 11:30:36 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:36 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:36.064525 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:36.054113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:53883)
0807 11:30:36.054735 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:36.054763 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:36.054923 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:36.055371 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:36.055376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:36.055883 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:36.056103 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:36.056116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.057397 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.057401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:36.058373 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:36.058385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.059198 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.059208 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:36.059470 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:36.060288 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:36.060312 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:36.064154 (+ 3842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":317,"threads_started":1}
W20260807 11:30:36.064779 3088 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:36.054190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:53883 (local address 127.24.103.126:32981)
0807 11:30:36.054766 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:36.054771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:36.054857 (+ 86us) server_negotiation.cc:407] Connection header received
0807 11:30:36.055086 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:36.055092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:36.055192 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:36.055344 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:36.056272 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.057246 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.058563 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.058983 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.059162 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:36.064467 (+ 5305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:53883: BlockingRecv error: recv got EOF from 127.24.103.67:53883 (error 108)
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":327,"threads_started":1}
Aug 07 11:30:36 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:36 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:36.069975 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:36.059083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:57671)
0807 11:30:36.059918 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:36.059940 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:36.060084 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:36.060558 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:36.060562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:36.060921 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:36.061130 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:36.061141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.062424 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.062433 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:36.064115 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:36.064134 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.065398 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.065403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:36.065539 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:36.066239 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:36.066259 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:36.069741 (+ 3482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":541,"threads_started":1}
W20260807 11:30:36.070327 3090 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:36.059146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:57671 (local address 127.24.103.126:32981)
0807 11:30:36.059744 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:36.059749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:36.060038 (+ 289us) server_negotiation.cc:407] Connection header received
0807 11:30:36.060266 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:36.060272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:36.060346 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:36.060467 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:36.061296 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.062267 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.064300 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.065245 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.065345 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:36.069988 (+ 4643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:57671: BlockingRecv error: recv got EOF from 127.24.103.65:57671 (error 108)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":297,"threads_started":1}
Aug 07 11:30:36 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:36 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:36.551450 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:36.536097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:57221)
0807 11:30:36.536633 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:36.536650 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:36.536786 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:36.537196 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:36.537200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:36.537611 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:36.537775 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:36.537785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.538884 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.538888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:36.539947 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:36.539961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.540638 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.540641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:36.540764 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:36.547653 (+ 6889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:36.547678 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:36.551112 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":322,"threads_started":1}
W20260807 11:30:36.551679 3090 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:36.536293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:57221 (local address 127.24.103.126:32981)
0807 11:30:36.536539 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:36.536545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:36.536744 (+ 199us) server_negotiation.cc:407] Connection header received
0807 11:30:36.536896 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:36.536899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:36.536964 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:36.537047 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:36.537941 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.538725 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.540095 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:36.540505 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:36.540601 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:36.551441 (+ 10840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:57221: BlockingRecv error: recv got EOF from 127.24.103.66:57221 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 07 11:30:37 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:37 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:37.076050 3092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:37.065650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:36067)
0807 11:30:37.066169 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:37.066190 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:37.066312 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:37.066792 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:37.066795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:37.067207 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:37.067360 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:37.067368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.068487 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.068491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:37.069493 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:37.069505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.070289 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.070295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:37.070495 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:37.071318 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:37.071353 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:37.075682 (+ 4329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":266,"threads_started":1}
W20260807 11:30:37.076236 3093 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:37.065731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:36067 (local address 127.24.103.126:32981)
0807 11:30:37.066529 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:37.066533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:37.066548 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:30:37.066594 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:37.066598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:37.066654 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:37.066742 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:37.067495 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.068357 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.069669 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.070134 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.070275 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:37.075884 (+ 5609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:36067: BlockingRecv error: recv got EOF from 127.24.103.67:36067 (error 108)
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":600,"threads_started":1}
Aug 07 11:30:37 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:37 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:37.082458 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:37.071088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:47027)
0807 11:30:37.073796 (+ 2708us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:37.073818 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:37.073935 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:37.074295 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:37.074298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:37.074842 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:37.075119 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:37.075135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.076498 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.076502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:37.077426 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:37.077437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.078189 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.078193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:37.078311 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:37.079104 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:37.079137 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:37.082099 (+ 2962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2559,"thread_start_us":2402,"threads_started":1}
W20260807 11:30:37.083112 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:37.071166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:47027 (local address 127.24.103.126:32981)
0807 11:30:37.073617 (+ 2451us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:37.073623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:37.073895 (+ 272us) server_negotiation.cc:407] Connection header received
0807 11:30:37.074027 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:37.074032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:37.074121 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:37.074237 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:37.075279 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.076228 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.077602 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.078030 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.078142 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:37.082712 (+ 4570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:47027: BlockingRecv error: recv got EOF from 127.24.103.65:47027 (error 108)
Metrics: {"server-negotiator.queue_time_us":2332,"thread_start_us":2116,"threads_started":1}
Aug 07 11:30:37 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:37 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:37.561399 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:37.552450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:46533)
0807 11:30:37.553073 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:37.553089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:37.553254 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:37.553619 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:37.553624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:37.554111 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:37.554301 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:37.554310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.555427 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.555434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:37.556357 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:37.556368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.557066 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.557072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:37.557259 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:37.557860 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:37.557877 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:37.561050 (+ 3173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":347,"threads_started":1}
W20260807 11:30:37.561604 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:37.552685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:46533 (local address 127.24.103.126:32981)
0807 11:30:37.552923 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:37.552929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:37.553181 (+ 252us) server_negotiation.cc:407] Connection header received
0807 11:30:37.553384 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:37.553392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:37.553460 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:37.553607 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:37.554452 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.555282 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.556504 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:37.556922 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:37.557055 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:37.561282 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:46533: BlockingRecv error: recv got EOF from 127.24.103.66:46533 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 07 11:30:38 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:38 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:38.086458 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:38.077131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:48119)
0807 11:30:38.077688 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:38.077719 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:38.077828 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:38.078222 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:38.078225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:38.078625 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:38.078809 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:38.078818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.079904 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.079909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:38.080890 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:38.080903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.081755 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.081760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:38.081877 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:38.082480 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:38.082499 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:38.086097 (+ 3598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":285,"threads_started":1}
W20260807 11:30:38.086663 3098 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:38.077330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:48119 (local address 127.24.103.126:32981)
0807 11:30:38.077965 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:38.077970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:38.077984 (+ 14us) server_negotiation.cc:407] Connection header received
0807 11:30:38.078028 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:38.078031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:38.078088 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:38.078174 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:38.078950 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.079749 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.081062 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.081620 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.081714 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:38.086382 (+ 4668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:48119: BlockingRecv error: recv got EOF from 127.24.103.67:48119 (error 108)
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":429,"threads_started":1}
Aug 07 11:30:38 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:38 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:38.093309 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:38.083416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:60321)
0807 11:30:38.084054 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:38.084080 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:38.084208 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:38.084701 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:38.084706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:38.085162 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:38.085410 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:38.085419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.086897 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.086903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:38.088166 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:38.088179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.089039 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.089044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:38.089167 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:38.089824 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:38.089842 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:38.092950 (+ 3108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":313,"threads_started":1}
W20260807 11:30:38.093521 3100 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:38.083465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:60321 (local address 127.24.103.126:32981)
0807 11:30:38.084409 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:38.084413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:38.084427 (+ 14us) server_negotiation.cc:407] Connection header received
0807 11:30:38.084467 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:38.084470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:38.084533 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:38.084619 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:38.085569 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.086750 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.088353 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.088842 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.088945 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:38.093200 (+ 4255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:60321: BlockingRecv error: recv got EOF from 127.24.103.65:60321 (error 108)
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":634,"threads_started":1}
Aug 07 11:30:38 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:38 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:38.572105 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:38.562707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:42779)
0807 11:30:38.563298 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:38.563315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:38.563442 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:38.563763 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:38.563766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:38.564209 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:38.564368 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:38.564377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.565575 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.565579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:38.566549 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:38.566565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.567420 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.567425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:38.567550 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:38.568260 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:38.568278 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:38.571846 (+ 3568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":365,"threads_started":1}
W20260807 11:30:38.572327 3100 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:38.562884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:42779 (local address 127.24.103.126:32981)
0807 11:30:38.563185 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:38.563193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:38.563422 (+ 229us) server_negotiation.cc:407] Connection header received
0807 11:30:38.563552 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:38.563556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:38.563623 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:38.563726 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:38.564562 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.565420 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.566710 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:38.567263 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:38.567374 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:38.572076 (+ 4702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:42779: BlockingRecv error: recv got EOF from 127.24.103.66:42779 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 07 11:30:39 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:39 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:39.098079 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:39.087933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:58121)
0807 11:30:39.088653 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:39.088688 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:39.088845 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:39.089274 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:39.089279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:39.089882 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:39.090045 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:39.090054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.091097 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.091101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:39.092321 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:39.092334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.093252 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.093257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:39.093387 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:39.094171 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:39.094200 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:39.097846 (+ 3646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":437,"threads_started":1}
W20260807 11:30:39.098424 3104 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:39.088117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:58121 (local address 127.24.103.126:32981)
0807 11:30:39.088632 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:39.088638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:39.088778 (+ 140us) server_negotiation.cc:407] Connection header received
0807 11:30:39.088994 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:39.088999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:39.089084 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:39.089190 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:39.090180 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.090910 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.092520 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.093080 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.093197 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:39.098103 (+ 4906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:58121: BlockingRecv error: recv got EOF from 127.24.103.67:58121 (error 108)
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":289,"threads_started":1}
Aug 07 11:30:39 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:39 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:39.104627 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:39.094460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:39179)
0807 11:30:39.095236 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:39.095252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:39.095404 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:39.095797 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:39.095801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:39.096263 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:39.096449 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:39.096460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.097865 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.097871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:39.099269 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:39.099284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.100101 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.100106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:39.100275 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:39.100966 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:39.100998 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:39.104338 (+ 3340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":458,"threads_started":1}
W20260807 11:30:39.104764 3105 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:39.094517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:39179 (local address 127.24.103.126:32981)
0807 11:30:39.095091 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:39.095097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:39.095334 (+ 237us) server_negotiation.cc:407] Connection header received
0807 11:30:39.095536 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:39.095540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:39.095615 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:39.095713 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:39.096622 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.097661 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.099423 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.099825 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.099910 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:39.104528 (+ 4618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:39179: BlockingRecv error: recv got EOF from 127.24.103.65:39179 (error 108)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":271,"threads_started":1}
Aug 07 11:30:39 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:39 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:39.581735 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:39.573034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:40805)
0807 11:30:39.573652 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:39.573673 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:39.573820 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:39.574128 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:39.574132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:39.574569 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:39.574742 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:39.574751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.576089 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.576093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:39.577169 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:39.577183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.577840 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.577844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:39.577955 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:39.578521 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:39.578540 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:39.581447 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":492,"threads_started":1}
W20260807 11:30:39.581889 3105 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:39.573191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:40805 (local address 127.24.103.126:32981)
0807 11:30:39.573424 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:39.573430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:39.573761 (+ 331us) server_negotiation.cc:407] Connection header received
0807 11:30:39.573923 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:39.573927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:39.573995 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:39.574079 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:39.574913 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.575905 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.577309 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:39.577696 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:39.577799 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:39.581649 (+ 3850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:40805: BlockingRecv error: recv got EOF from 127.24.103.66:40805 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 07 11:30:40 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:40 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:40.110006 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:40.099275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:58421)
0807 11:30:40.100520 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:40.100542 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:40.100711 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:40.101159 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:40.101169 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:40.101862 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:40.102104 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:40.102122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.103458 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.103462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:40.104518 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:40.104539 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.105355 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.105362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:40.105532 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:40.106492 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:40.106522 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:40.109629 (+ 3107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1112,"thread_start_us":899,"threads_started":1}
W20260807 11:30:40.110210 3112 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:40.099362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:58421 (local address 127.24.103.126:32981)
0807 11:30:40.100067 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:40.100077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:40.100636 (+ 559us) server_negotiation.cc:407] Connection header received
0807 11:30:40.100859 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:40.100864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:40.100974 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:40.101081 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:40.102348 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.103284 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.104677 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.105150 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.105387 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:40.109861 (+ 4474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:58421: BlockingRecv error: recv got EOF from 127.24.103.67:58421 (error 108)
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":359,"threads_started":1}
Aug 07 11:30:40 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:40 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:40.115299 3113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:40.105675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:38987)
0807 11:30:40.106261 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:40.106283 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:40.106418 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:40.106832 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:40.106837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:40.107395 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:40.107551 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:40.107560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.108710 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.108724 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:40.109684 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:40.109695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.110762 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.110770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:40.110940 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:40.111565 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:40.111584 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:40.115033 (+ 3449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":307,"threads_started":1}
W20260807 11:30:40.115653 3114 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:40.105867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:38987 (local address 127.24.103.126:32981)
0807 11:30:40.106556 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:40.106569 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:40.106583 (+ 14us) server_negotiation.cc:407] Connection header received
0807 11:30:40.106620 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:40.106623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:40.106674 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:40.106787 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:40.107715 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.108559 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.109844 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.110552 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.110689 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:40.115316 (+ 4627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:38987: BlockingRecv error: recv got EOF from 127.24.103.65:38987 (error 108)
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":435,"threads_started":1}
Aug 07 11:30:40 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:40 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:40.592180 3115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:40.582927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:49887)
0807 11:30:40.583605 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:40.583622 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:40.583756 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:40.584083 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:40.584086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:40.584508 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:40.584665 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:40.584673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.585994 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.585999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:40.586991 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:40.587004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.587918 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.587922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:40.588037 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:40.588783 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:40.588801 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:40.591910 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":422,"threads_started":1}
W20260807 11:30:40.592454 3114 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:40.583102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:49887 (local address 127.24.103.126:32981)
0807 11:30:40.583301 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:40.583305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:40.583729 (+ 424us) server_negotiation.cc:407] Connection header received
0807 11:30:40.583876 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:40.583880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:40.583949 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:40.584034 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:40.584885 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.585837 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.587236 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:40.587679 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:40.587843 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:40.592185 (+ 4342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:49887: BlockingRecv error: recv got EOF from 127.24.103.66:49887 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 07 11:30:41 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:41 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:41.122679 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:41.111208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:39363)
0807 11:30:41.111867 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:41.111890 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:41.112038 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:41.112536 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:41.112540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:41.113032 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:41.113215 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:41.113225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.114371 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.114376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:41.115422 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:41.115434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.116642 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.116655 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:41.116922 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:41.118180 (+ 1258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:41.118209 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:41.122324 (+ 4115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":324,"threads_started":1}
W20260807 11:30:41.123592 3117 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:41.111285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:39363 (local address 127.24.103.126:32981)
0807 11:30:41.111838 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:41.111842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:41.111973 (+ 131us) server_negotiation.cc:407] Connection header received
0807 11:30:41.112280 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:41.112285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:41.112373 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:41.112477 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:41.113379 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.114209 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.115569 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.116379 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.116524 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:41.122890 (+ 6366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:39363: BlockingRecv error: recv got EOF from 127.24.103.67:39363 (error 108)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":384,"threads_started":1}
Aug 07 11:30:41 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:41 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:41.128175 3119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:41.116379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:39923)
0807 11:30:41.117547 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:41.117572 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:41.117757 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:41.118527 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:41.118533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:41.119102 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:41.119355 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:41.119369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.121069 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.121082 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:41.122662 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:41.122684 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.123876 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.123883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:41.124081 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:41.124641 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:41.124660 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:41.127922 (+ 3262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":791,"threads_started":1}
W20260807 11:30:41.128676 3120 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:41.116490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:39923 (local address 127.24.103.126:32981)
0807 11:30:41.117231 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:41.117237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:41.117905 (+ 668us) server_negotiation.cc:407] Connection header received
0807 11:30:41.117973 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:41.117979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:41.118073 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:41.118207 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:41.119518 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.120712 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.122880 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.123715 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.123851 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:41.128284 (+ 4433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:39923: BlockingRecv error: recv got EOF from 127.24.103.65:39923 (error 108)
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":388,"threads_started":1}
Aug 07 11:30:41 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:41 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:41.606971 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:41.593260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:32859)
0807 11:30:41.593862 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:41.593878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:41.593998 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:41.594401 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:41.594405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:41.594814 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:41.594983 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:41.594992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.596116 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.596125 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:41.597334 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:41.597374 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.598379 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.598386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:41.598627 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:41.603189 (+ 4562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:41.603216 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:41.606575 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":345,"threads_started":1}
W20260807 11:30:41.607272 3120 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:41.593292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:32859 (local address 127.24.103.126:32981)
0807 11:30:41.593466 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:41.593470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:41.593985 (+ 515us) server_negotiation.cc:407] Connection header received
0807 11:30:41.594144 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:41.594149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:41.594214 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:41.594303 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:41.595202 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.595947 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.597542 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:41.598198 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:41.598299 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:41.606953 (+ 8654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:32859: BlockingRecv error: recv got EOF from 127.24.103.66:32859 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 07 11:30:42 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:42 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:42.134938 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:42.123977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:33699)
0807 11:30:42.124652 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:42.124674 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:42.124799 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:42.125173 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:42.125178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:42.125607 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:42.125827 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:42.125837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.127483 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.127488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:42.128885 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:42.128899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.130026 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.130037 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:42.130280 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:42.130970 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:42.130997 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:42.134546 (+ 3549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":384,"threads_started":1}
W20260807 11:30:42.135162 3124 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:42.124121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:33699 (local address 127.24.103.126:32981)
0807 11:30:42.124621 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:42.124625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:42.124802 (+ 177us) server_negotiation.cc:407] Connection header received
0807 11:30:42.124888 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:42.124892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:42.124970 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:42.125070 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:42.126060 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.127296 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.129041 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.129562 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.129714 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:42.134792 (+ 5078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:33699: BlockingRecv error: recv got EOF from 127.24.103.67:33699 (error 108)
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":276,"threads_started":1}
Aug 07 11:30:42 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:42 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:42.139492 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:42.129378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:47595)
0807 11:30:42.129980 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:42.130003 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:42.130137 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:42.131314 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:42.131317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:42.131704 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:42.131845 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:42.131855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.133222 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.133226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:42.134331 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:42.134344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.135476 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.135481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:42.135608 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:42.136216 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:42.136235 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:42.139085 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":345,"threads_started":1}
W20260807 11:30:42.139712 3128 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:42.129460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:47595 (local address 127.24.103.126:32981)
0807 11:30:42.130915 (+ 1455us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:42.130923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:42.130942 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:30:42.131001 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:42.131007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:42.131126 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:42.131243 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:42.132046 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.133023 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.134475 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.134945 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.135450 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:42.139353 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:47595: BlockingRecv error: recv got EOF from 127.24.103.65:47595 (error 108)
Metrics: {"server-negotiator.queue_time_us":1288,"thread_start_us":1177,"threads_started":1}
Aug 07 11:30:42 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:42 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:42.618748 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:42.608234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:60581)
0807 11:30:42.608950 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:42.608969 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:42.609141 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:42.609561 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:42.609566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:42.610020 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:42.610216 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:42.610226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.611542 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.611560 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:42.612657 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:42.612670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.613507 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.613513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:42.613707 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:42.614514 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:42.614534 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:42.618373 (+ 3839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":476,"threads_started":1}
W20260807 11:30:42.618871 3128 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:42.608245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:60581 (local address 127.24.103.126:32981)
0807 11:30:42.608453 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:42.608461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:42.609069 (+ 608us) server_negotiation.cc:407] Connection header received
0807 11:30:42.609315 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:42.609321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:42.609405 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:42.609527 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:42.610402 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.611372 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.612832 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:42.613309 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:42.613462 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:42.618600 (+ 5138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:60581: BlockingRecv error: recv got EOF from 127.24.103.66:60581 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 07 11:30:43 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:43 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:43.145522 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:43.136233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:58495)
0807 11:30:43.136987 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:43.137015 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:43.137185 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:43.137560 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:43.137564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:43.137976 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:43.138131 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:43.138140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.139212 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.139219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:43.140173 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:43.140185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.141318 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.141325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:43.141505 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:43.142192 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:43.142210 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:43.145171 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":304,"threads_started":1}
W20260807 11:30:43.145684 3132 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:43.136422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:58495 (local address 127.24.103.126:32981)
0807 11:30:43.137001 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:43.137009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:43.137115 (+ 106us) server_negotiation.cc:407] Connection header received
0807 11:30:43.137317 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:43.137323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:43.137407 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:43.137526 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:43.138294 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.139002 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.140327 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.141113 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.141262 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:43.145354 (+ 4092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:58495: BlockingRecv error: recv got EOF from 127.24.103.67:58495 (error 108)
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":331,"threads_started":1}
Aug 07 11:30:43 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:43 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:43.149343 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:43.140530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:38579)
0807 11:30:43.141081 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:43.141108 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:43.141241 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:43.141756 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:43.141762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:43.142202 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:43.142376 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:43.142386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.143493 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.143499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:43.144459 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:43.144472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.145195 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.145201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:43.145393 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:43.146076 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:43.146094 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:43.149001 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":277,"threads_started":1}
W20260807 11:30:43.150008 3134 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:43.140610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:38579 (local address 127.24.103.126:32981)
0807 11:30:43.141445 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:43.141449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:43.141463 (+ 14us) server_negotiation.cc:407] Connection header received
0807 11:30:43.141506 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:43.141509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:43.141562 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:43.141679 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:43.142534 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.143355 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.144614 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.145020 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.145168 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:43.149354 (+ 4186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:38579: BlockingRecv error: recv got EOF from 127.24.103.65:38579 (error 108)
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":533,"threads_started":1}
Aug 07 11:30:43 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:43 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:43.629846 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:43.620238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:48663)
0807 11:30:43.620864 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:43.620881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:43.621051 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:43.621462 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:43.621467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:43.621963 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:43.622181 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:43.622192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.623318 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.623325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:43.624544 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:43.624581 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.625528 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.625534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:43.625758 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:43.626456 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:43.626475 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:43.629476 (+ 3001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":329,"threads_started":1}
W20260807 11:30:43.629935 3134 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:43.620459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:48663 (local address 127.24.103.126:32981)
0807 11:30:43.620738 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:43.620747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:43.620984 (+ 237us) server_negotiation.cc:407] Connection header received
0807 11:30:43.621209 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:43.621221 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:43.621287 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:43.621435 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:43.622353 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.623187 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.624740 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:43.625357 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:43.625503 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:43.629700 (+ 4197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:48663: BlockingRecv error: recv got EOF from 127.24.103.66:48663 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Aug 07 11:30:44 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:44 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:44.158268 3136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:44.146691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:40381)
0807 11:30:44.147615 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:44.147639 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:44.147817 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:44.148308 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:44.148312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:44.148826 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:44.149014 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:44.149023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.150054 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.150060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:44.152049 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:44.152067 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.152986 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.152991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:44.153169 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:44.153868 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:44.153887 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:44.157894 (+ 4007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":648,"threads_started":1}
W20260807 11:30:44.158370 3137 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:44.146764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:40381 (local address 127.24.103.126:32981)
0807 11:30:44.147737 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:44.147741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:44.147759 (+ 18us) server_negotiation.cc:407] Connection header received
0807 11:30:44.147974 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:44.147980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:44.148118 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:44.148211 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:44.149163 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.149907 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.152268 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.152810 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.152968 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:44.158096 (+ 5128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:40381: BlockingRecv error: recv got EOF from 127.24.103.67:40381 (error 108)
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":483,"threads_started":1}
Aug 07 11:30:44 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:44 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:44.161123 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:44.150503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:35857)
0807 11:30:44.151057 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:44.151080 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:44.151234 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:44.151764 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:44.151771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:44.152351 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:44.152511 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:44.152521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.154004 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.154013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:44.155646 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:44.155660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.156680 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.156689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:44.156907 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:44.157551 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:44.157591 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:44.160897 (+ 3306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":297,"threads_started":1}
W20260807 11:30:44.161410 3139 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:44.150586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:35857 (local address 127.24.103.126:32981)
0807 11:30:44.151462 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:44.151466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:44.151485 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:30:44.151526 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:44.151529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:44.151583 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:44.151666 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:44.152678 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.153848 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.155796 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.156463 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.156614 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:44.161135 (+ 4521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:35857: BlockingRecv error: recv got EOF from 127.24.103.65:35857 (error 108)
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":646,"threads_started":1}
Aug 07 11:30:44 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:44 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:44.640378 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:44.631144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:48263)
0807 11:30:44.631677 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:44.631695 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:44.631888 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:44.632314 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:44.632319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:44.632879 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:44.633040 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:44.633049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.634203 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.634212 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:44.635243 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:44.635255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.636092 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.636098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:44.636301 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:44.637052 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:44.637080 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:44.640134 (+ 3054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":305,"threads_started":1}
W20260807 11:30:44.640713 3139 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:44.631192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:48263 (local address 127.24.103.126:32981)
0807 11:30:44.631438 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:44.631447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:44.631797 (+ 350us) server_negotiation.cc:407] Connection header received
0807 11:30:44.632054 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:44.632059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:44.632124 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:44.632239 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:44.633210 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.634045 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.635391 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:44.635927 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:44.636076 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:44.640416 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:48263: BlockingRecv error: recv got EOF from 127.24.103.66:48263 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 07 11:30:45 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:45 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:45 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:45.170539 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:45.159600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:55245)
0807 11:30:45.160238 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:45.160264 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:45.160385 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:45.160843 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:45.160847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:45.161342 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:45.161510 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:45.161519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.163104 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.163112 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:45.164463 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:45.164482 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.165676 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.165683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:45.165907 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:45.166730 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:45.166751 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:45.170170 (+ 3419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":342,"threads_started":1}
Aug 07 11:30:45 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:30:45.170742 3144 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:45.159682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:55245 (local address 127.24.103.126:32981)
0807 11:30:45.160523 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:45.160527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:45.160543 (+ 16us) server_negotiation.cc:407] Connection header received
0807 11:30:45.160590 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:45.160593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:45.160652 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:45.160748 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:45.161694 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.162902 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.164643 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.165515 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.165670 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:45.170404 (+ 4734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:55245: BlockingRecv error: recv got EOF from 127.24.103.67:55245 (error 108)
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":743,"thread_start_us":642,"threads_started":1}
I20260807 11:30:45.171880 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:45.162094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:39265)
0807 11:30:45.162988 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:45.163004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:45.163189 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:45.163555 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:45.163561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:45.163976 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:45.164206 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:45.164224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.165823 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.165828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:45.166830 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:45.166843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.167660 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.167667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:45.167842 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:45.168492 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:45.168509 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:45.171511 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":619,"threads_started":1}
W20260807 11:30:45.172048 3146 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:45.162229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:39265 (local address 127.24.103.126:32981)
0807 11:30:45.162879 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:45.162885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:45.163129 (+ 244us) server_negotiation.cc:407] Connection header received
0807 11:30:45.163326 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:45.163332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:45.163399 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:45.163512 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:45.164366 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.165465 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.166989 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.167456 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.167593 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:45.171722 (+ 4129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:39265: BlockingRecv error: recv got EOF from 127.24.103.65:39265 (error 108)
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":278,"threads_started":1}
Aug 07 11:30:45 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:45 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:45.650935 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:45.641710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:33987)
0807 11:30:45.642386 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:45.642404 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:45.642537 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:45.642955 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:45.642959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:45.643455 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:45.643618 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:45.643626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.644988 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.644994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:45.645931 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:45.645943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.646700 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.646707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:45.646875 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:45.647506 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:45.647524 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:45.650531 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":345,"threads_started":1}
W20260807 11:30:45.651036 3146 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:45.641999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:33987 (local address 127.24.103.126:32981)
0807 11:30:45.642266 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:45.642276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:45.642499 (+ 223us) server_negotiation.cc:407] Connection header received
0807 11:30:45.642701 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:45.642707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:45.642775 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:45.642869 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:45.643835 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.644821 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.646093 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:45.646496 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:45.646638 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:45.650755 (+ 4117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:33987: BlockingRecv error: recv got EOF from 127.24.103.66:33987 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 07 11:30:46 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:46 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:46 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:46.181902 3148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:46.171832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:33525)
0807 11:30:46.172492 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:46.172519 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:46.172666 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:46.173121 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:46.173126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:46.173760 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:46.173949 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:46.173958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.175089 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.175094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:46.176246 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:46.176263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.177215 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.177222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:46.177408 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:46.178187 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:46.178207 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:46.181584 (+ 3377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":315,"threads_started":1}
Aug 07 11:30:46 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:30:46.182173 3149 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:46.171930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:33525 (local address 127.24.103.126:32981)
0807 11:30:46.172741 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:46.172746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:46.172786 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:30:46.172849 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:46.172856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:46.172950 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:46.173069 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:46.174135 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.174889 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.176415 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.177009 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.177133 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:46.181893 (+ 4760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:33525: BlockingRecv error: recv got EOF from 127.24.103.67:33525 (error 108)
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":602,"threads_started":1}
I20260807 11:30:46.183185 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:46.172909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:41991)
0807 11:30:46.173536 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:46.173565 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:46.173689 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:46.174216 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:46.174224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:46.174635 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:46.174808 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:46.174817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.176344 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.176351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:46.177824 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:46.177840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.178565 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.178571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:46.178755 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:46.179480 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:46.179504 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:46.182883 (+ 3379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":391,"threads_started":1}
W20260807 11:30:46.183379 3151 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:46.172974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:41991 (local address 127.24.103.126:32981)
0807 11:30:46.173933 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:46.173937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:46.173952 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:30:46.173990 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:46.173994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:46.174040 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:46.174175 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:46.174965 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.176018 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.177974 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.178406 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.178544 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:46.183116 (+ 4572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:41991: BlockingRecv error: recv got EOF from 127.24.103.65:41991 (error 108)
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":619,"threads_started":1}
Aug 07 11:30:46 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:46 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:46.668280 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:46.652206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:56809)
0807 11:30:46.652814 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:46.652844 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:46.652975 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:46.653655 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:46.653659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:46.654092 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:46.654281 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:46.654290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.655996 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.656000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:46.657238 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:46.657253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.658086 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.658091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:46.658218 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:46.664343 (+ 6125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:46.664370 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:46.667887 (+ 3517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":365,"threads_started":1}
W20260807 11:30:46.668424 3151 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:46.652345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:56809 (local address 127.24.103.126:32981)
0807 11:30:46.652549 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:46.652557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:46.652945 (+ 388us) server_negotiation.cc:407] Connection header received
0807 11:30:46.653125 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:46.653130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:46.653210 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:46.653303 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:46.654459 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.655719 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.657412 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:46.657876 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:46.657982 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:46.668163 (+ 10181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:56809: BlockingRecv error: recv got EOF from 127.24.103.66:56809 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 07 11:30:47 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:47 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.67@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:47 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:47.194428 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:47.183210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.67:36375)
0807 11:30:47.183883 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:47.183916 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:47.184065 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:47.184662 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:47.184671 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:47.185312 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:47.185510 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:47.185520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.186987 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.186993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:47.188605 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:47.188622 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.189564 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.189571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:47.189767 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:47.190615 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:47.190633 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:47.194059 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":346,"threads_started":1}
Aug 07 11:30:47 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.65@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 11:30:47.194859 3154 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:47.183465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:36375 (local address 127.24.103.126:32981)
0807 11:30:47.184236 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:47.184240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:47.184257 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:30:47.184338 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:47.184344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:47.184440 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:47.184568 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:47.185622 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.186800 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.188779 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.189414 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.189509 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:47.194331 (+ 4822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.67:36375: BlockingRecv error: recv got EOF from 127.24.103.67:36375 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":672,"thread_start_us":558,"threads_started":1}
I20260807 11:30:47.195962 3155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:47.184355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.65:52455)
0807 11:30:47.185420 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:47.185435 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:47.185599 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:47.185967 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:47.185972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:47.186482 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:47.186685 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:47.186697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.188677 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.188684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:47.190250 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:47.190265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.191058 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.191065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:47.191284 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:47.191924 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:47.191942 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:47.195667 (+ 3725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":803,"threads_started":1}
W20260807 11:30:47.196161 3156 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:47.184415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:52455 (local address 127.24.103.126:32981)
0807 11:30:47.185230 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:47.185237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:47.185540 (+ 303us) server_negotiation.cc:407] Connection header received
0807 11:30:47.185746 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:47.185750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:47.185808 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:47.185890 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:47.186860 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.188102 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.190396 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.190882 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.191020 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:47.195905 (+ 4885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.65:52455: BlockingRecv error: recv got EOF from 127.24.103.65:52455 (error 108)
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":510,"threads_started":1}
Aug 07 11:30:47 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 11:30:47 dist-test-slave-wl2h krb5kdc[1964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.103.66@KRBTEST.COM for oryx/127.24.103.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 11:30:47.678782 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:47.669634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:32981 (local address 127.24.103.66:52945)
0807 11:30:47.670268 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:47.670284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:47.670447 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:47.670832 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:47.670837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:47.671339 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:47.671507 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:47.671516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.672780 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.672785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:47.673715 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:47.673726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.674448 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.674454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:47.674630 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:47.675313 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:47.675331 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:47.678403 (+ 3072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.103.126:32981: Server oryx/127.24.103.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":312,"threads_started":1}
W20260807 11:30:47.678884 3156 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:47.669786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:52945 (local address 127.24.103.126:32981)
0807 11:30:47.670032 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:47.670038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:47.670376 (+ 338us) server_negotiation.cc:407] Connection header received
0807 11:30:47.670592 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:47.670598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:47.670673 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:47.670780 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:47.671657 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.672653 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.673859 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:47.674297 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:47.674431 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:47.678649 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.103.66:52945: BlockingRecv error: recv got EOF from 127.24.103.66:52945 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:30:48.001905 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 2613
I20260807 11:30:48.016466 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:48.003245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.24.103.65:42969)
0807 11:30:48.003874 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:48.003882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:48.003913 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:30:48.003978 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:48.003985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:48.004214 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:48.004389 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:48.006121 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:48.007554 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:48.009046 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:48.009637 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:48.009750 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:48.012958 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:48.012968 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:48.012982 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:48.013021 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:48.015897 (+ 2876us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.103.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 11:30:48.016224 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37712: Request ticket server kudu/127.24.103.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":321,"threads_started":1}
W20260807 11:30:48.016597 3159 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37712: Request ticket server kudu/127.24.103.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 11:30:48.016801 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 2613: Not authorized: Client connection negotiation failed: client connection to 127.24.103.65:42969: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.103.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 11:30:48.016995 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 2613
I20260807 11:30:48.040016 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 2781
I20260807 11:30:48.053411 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:48.041194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.24.103.66:44355)
0807 11:30:48.041879 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:48.041886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:48.041919 (+ 33us) server_negotiation.cc:407] Connection header received
0807 11:30:48.041978 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:48.041984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:48.042217 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:48.042523 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:48.043991 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:48.045073 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:48.046324 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:48.046901 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:48.047063 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:48.049815 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:48.049824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:48.049839 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:48.049903 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:48.052664 (+ 2761us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.103.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 11:30:48.053037 (+ 373us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57664: Request ticket server kudu/127.24.103.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":345,"threads_started":1}
W20260807 11:30:48.053505 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 2781: Not authorized: Client connection negotiation failed: client connection to 127.24.103.66:44355: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.103.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 11:30:48.053552 3160 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57664: Request ticket server kudu/127.24.103.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 11:30:48.053718 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 2781
I20260807 11:30:48.075611 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 2923
I20260807 11:30:48.088995 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:48.076884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.24.103.67:46589)
0807 11:30:48.077543 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:48.077550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:48.077577 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:30:48.077648 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:48.077655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:48.077875 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:48.078042 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:48.079525 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:48.080597 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:48.081827 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:48.082426 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:48.082530 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:48.085286 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:48.085296 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:48.085310 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:48.085350 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:48.088444 (+ 3094us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.103.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 11:30:48.088719 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39484: Request ticket server kudu/127.24.103.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":361,"threads_started":1}
W20260807 11:30:48.089108 3161 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39484: Request ticket server kudu/127.24.103.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 11:30:48.089200 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 2923: Not authorized: Client connection negotiation failed: client connection to 127.24.103.67:46589: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.103.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 11:30:48.089347 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 2923
I20260807 11:30:48.113013 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 2534
I20260807 11:30:48.114449 2604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:48.240605 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 2534
2026-08-07T11:30:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21809 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T11:30:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:30:48Z Disabled control of system clock
I20260807 11:30:48.339974 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38327
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 11:30:48.695613 3167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:48.695941 3167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:48.696142 3167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:48.703809 3167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:48.703928 3167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:48.704001 3167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:48.704089 3167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:48.704164 3167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:30:48.713824 3167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:48.715345 3167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:48.717356 3167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:48.730122 3172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:48.730028 3176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:48.730139 3173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:48.730919 3167 server_base.cc:1034] running on GCE node
I20260807 11:30:48.731813 3167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:48.733337 3167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:48.734601 3167 hybrid_clock.cc:648] HybridClock initialized: now 1786102248734548 us; error 47 us; skew 500 ppm
I20260807 11:30:48.735064 3167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:30:48.737695 3167 webserver.cc:460] Webserver started at http://127.24.103.126:45049/ using document root <none> and password file <none>
I20260807 11:30:48.738468 3167 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:48.738615 3167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:48.738969 3167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:48.742142 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "7f33721ec9fd4ed8ae84de7e26b511bb"
format_stamp: "Formatted at 2026-08-07 11:30:48 on dist-test-slave-wl2h"
server_key: "5003661103d9024866069cc06c47a8e1"
server_key_iv: "8b982ffdfd704d54eb6b0f24d72cb2f7"
server_key_version: "encryptionkey@0"
I20260807 11:30:48.742992 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "7f33721ec9fd4ed8ae84de7e26b511bb"
format_stamp: "Formatted at 2026-08-07 11:30:48 on dist-test-slave-wl2h"
server_key: "5003661103d9024866069cc06c47a8e1"
server_key_iv: "8b982ffdfd704d54eb6b0f24d72cb2f7"
server_key_version: "encryptionkey@0"
I20260807 11:30:48.748500 3167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 11:30:48.752135 3185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:48.753641 3167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:30:48.753852 3167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "7f33721ec9fd4ed8ae84de7e26b511bb"
format_stamp: "Formatted at 2026-08-07 11:30:48 on dist-test-slave-wl2h"
server_key: "5003661103d9024866069cc06c47a8e1"
server_key_iv: "8b982ffdfd704d54eb6b0f24d72cb2f7"
server_key_version: "encryptionkey@0"
I20260807 11:30:48.754092 3167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:48.772792 3167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:48.773733 3167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:48.774044 3167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:48.794121 3167 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38327
I20260807 11:30:48.794133 3244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38327 every 8 connection(s)
I20260807 11:30:48.796182 3167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:30:48.800307 3245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:48.801438 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 3167
I20260807 11:30:48.801693 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:30:48.802383 24989 external_mini_cluster.cc:1428] Setting key 7a294c3b29f328624c2cb6ea466d82cb
I20260807 11:30:48.814524 3245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb: Bootstrap starting.
I20260807 11:30:48.816854 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:48.804453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.24.103.126:38327)
0807 11:30:48.805538 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:48.805551 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:48.805592 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:30:48.807252 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:48.807278 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:48.807733 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:48.808057 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:48.808804 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:48.810233 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:48.812222 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:48.812916 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:48.813050 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:48.814378 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:48.814385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:48.814399 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:48.815059 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:48.815169 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:48.815192 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:48.815285 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:48.815525 (+ 240us) server_negotiation.cc:299] Negotiation successful
0807 11:30:48.815880 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":419,"threads_started":1}
I20260807 11:30:48.818365 3245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:48.819535 3245 log.cc:824] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:48.822577 3245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb: No bootstrap required, opened a new log
I20260807 11:30:48.830407 3245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f33721ec9fd4ed8ae84de7e26b511bb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38327 } }
I20260807 11:30:48.830806 3245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:48.830906 3245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f33721ec9fd4ed8ae84de7e26b511bb, State: Initialized, Role: FOLLOWER
I20260807 11:30:48.831537 3245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f33721ec9fd4ed8ae84de7e26b511bb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38327 } }
I20260807 11:30:48.831753 3245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:30:48.831857 3245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:30:48.831988 3245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:48.834172 3245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f33721ec9fd4ed8ae84de7e26b511bb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38327 } }
I20260807 11:30:48.834750 3245 leader_election.cc:302] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f33721ec9fd4ed8ae84de7e26b511bb; no voters:
I20260807 11:30:48.835541 3245 leader_election.cc:288] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:30:48.835677 3251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:48.836401 3251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [term 1 LEADER]: Becoming Leader. State: Replica: 7f33721ec9fd4ed8ae84de7e26b511bb, State: Running, Role: LEADER
I20260807 11:30:48.837028 3251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f33721ec9fd4ed8ae84de7e26b511bb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38327 } }
I20260807 11:30:48.837929 3245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:48.840054 3252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f33721ec9fd4ed8ae84de7e26b511bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f33721ec9fd4ed8ae84de7e26b511bb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38327 } } }
I20260807 11:30:48.840144 3253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f33721ec9fd4ed8ae84de7e26b511bb. Latest consensus state: current_term: 1 leader_uuid: "7f33721ec9fd4ed8ae84de7e26b511bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f33721ec9fd4ed8ae84de7e26b511bb" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38327 } } }
I20260807 11:30:48.840579 3253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:48.840593 3252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:48.842535 3258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:30:48.847664 3258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:30:48.856917 3258 catalog_manager.cc:1347] Generated new cluster ID: 024c65a6c914478c97771beb75723d8a
I20260807 11:30:48.857051 3258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:30:48.879834 3258 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:30:48.881242 3258 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:30:48.890865 3258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f33721ec9fd4ed8ae84de7e26b511bb: Generated new TSK 0
I20260807 11:30:48.891826 3258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:30:48.904819 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38327
--builtin_ntp_servers=127.24.103.84:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 11:30:49.260411 3272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:49.260694 3272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:49.260846 3272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:49.268270 3272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:49.268379 3272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:49.268553 3272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:30:49.278234 3272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:49.279680 3272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:49.281559 3272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:49.295236 3283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:49.295289 3281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:49.295282 3280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:49.296387 3272 server_base.cc:1034] running on GCE node
I20260807 11:30:49.297019 3272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:49.298005 3272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:49.299226 3272 hybrid_clock.cc:648] HybridClock initialized: now 1786102249299173 us; error 32 us; skew 500 ppm
I20260807 11:30:49.299624 3272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:30:49.302096 3272 webserver.cc:460] Webserver started at http://127.24.103.65:40893/ using document root <none> and password file <none>
I20260807 11:30:49.302856 3272 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:49.302980 3272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:49.303395 3272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:49.306584 3272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "3918d65ecb7445c1b4c769fcb4a923f6"
format_stamp: "Formatted at 2026-08-07 11:30:49 on dist-test-slave-wl2h"
server_key: "fd32515b2f912dea8a8408c3ad6f2024"
server_key_iv: "d73e788881f74e3a8bb33dae8dddd854"
server_key_version: "encryptionkey@0"
I20260807 11:30:49.307521 3272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "3918d65ecb7445c1b4c769fcb4a923f6"
format_stamp: "Formatted at 2026-08-07 11:30:49 on dist-test-slave-wl2h"
server_key: "fd32515b2f912dea8a8408c3ad6f2024"
server_key_iv: "d73e788881f74e3a8bb33dae8dddd854"
server_key_version: "encryptionkey@0"
I20260807 11:30:49.313148 3272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260807 11:30:49.317118 3290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:49.318670 3272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:30:49.318846 3272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "3918d65ecb7445c1b4c769fcb4a923f6"
format_stamp: "Formatted at 2026-08-07 11:30:49 on dist-test-slave-wl2h"
server_key: "fd32515b2f912dea8a8408c3ad6f2024"
server_key_iv: "d73e788881f74e3a8bb33dae8dddd854"
server_key_version: "encryptionkey@0"
I20260807 11:30:49.319123 3272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:49.351372 3272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:49.352324 3272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:49.352648 3272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:49.353686 3272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:49.355264 3272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:49.355376 3272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:49.355512 3272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:49.355618 3272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:49.394387 3272 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:35057
I20260807 11:30:49.394484 3415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:35057 every 8 connection(s)
I20260807 11:30:49.395771 3272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:30:49.397604 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 3272
I20260807 11:30:49.397821 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:30:49.398711 24989 external_mini_cluster.cc:1428] Setting key d7187b7105bb07c0a0ae22e987450a0e
I20260807 11:30:49.401829 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38327
--builtin_ntp_servers=127.24.103.84:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 11:30:49.409099 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:49.398515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:34565 (local address 127.24.103.126:38327)
0807 11:30:49.398982 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:49.398987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:49.400556 (+ 1569us) server_negotiation.cc:407] Connection header received
0807 11:30:49.402554 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:49.402559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:49.402641 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:49.402781 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:49.403870 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:49.404589 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:49.406151 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:49.406567 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:49.406720 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:49.408192 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:49.408198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:49.408205 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:49.408230 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:49.408273 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:49.408286 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:49.408383 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:49.408633 (+ 250us) server_negotiation.cc:299] Negotiation successful
0807 11:30:49.408744 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":270,"threads_started":1}
I20260807 11:30:49.410111 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:49.399283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38327 (local address 127.24.103.65:34565)
0807 11:30:49.400389 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:49.400442 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:49.402295 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:49.402999 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:49.403007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:49.403147 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:49.403701 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:49.403717 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:49.404746 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:49.404757 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:49.405990 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:49.406002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:49.406799 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:49.406807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:49.407117 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:49.407869 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:30:49.407872 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:49.407911 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:30:49.407986 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:49.408385 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:49.408399 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:49.408408 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 11:30:49.408591 (+ 183us) client_negotiation.cc:240] Negotiation successful
0807 11:30:49.408979 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":349,"threads_started":1}
I20260807 11:30:49.412082 3416 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38327
I20260807 11:30:49.412400 3416 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:49.413415 3416 heartbeater.cc:511] Master 127.24.103.126:38327 requested a full tablet report, sending...
I20260807 11:30:49.415872 3206 ts_manager.cc:194] Registered new tserver with Master: 3918d65ecb7445c1b4c769fcb4a923f6 (127.24.103.65:35057)
I20260807 11:30:49.417392 3206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.65:34565
W20260807 11:30:49.756024 3420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:49.756423 3420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:49.756587 3420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:49.764132 3420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:49.764278 3420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:49.764462 3420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:30:49.773749 3420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:49.775130 3420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:49.776971 3420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:49.790668 3427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:49.790758 3426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:49.790758 3430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:49.792268 3420 server_base.cc:1034] running on GCE node
I20260807 11:30:49.792850 3420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:49.794085 3420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:49.795331 3420 hybrid_clock.cc:648] HybridClock initialized: now 1786102249795284 us; error 89 us; skew 500 ppm
I20260807 11:30:49.795708 3420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:30:49.798573 3420 webserver.cc:460] Webserver started at http://127.24.103.66:41525/ using document root <none> and password file <none>
I20260807 11:30:49.799523 3420 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:49.799645 3420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:49.799998 3420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:49.803854 3420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "592de1aad6604fa689e68de4f892dcb8"
format_stamp: "Formatted at 2026-08-07 11:30:49 on dist-test-slave-wl2h"
server_key: "a3239e5e2e79fda2a353d2a29122c1dc"
server_key_iv: "f73980e70995614e9fb166613617942a"
server_key_version: "encryptionkey@0"
I20260807 11:30:49.804839 3420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "592de1aad6604fa689e68de4f892dcb8"
format_stamp: "Formatted at 2026-08-07 11:30:49 on dist-test-slave-wl2h"
server_key: "a3239e5e2e79fda2a353d2a29122c1dc"
server_key_iv: "f73980e70995614e9fb166613617942a"
server_key_version: "encryptionkey@0"
I20260807 11:30:49.811530 3420 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 11:30:49.816021 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:49.817590 3420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:30:49.817740 3420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "592de1aad6604fa689e68de4f892dcb8"
format_stamp: "Formatted at 2026-08-07 11:30:49 on dist-test-slave-wl2h"
server_key: "a3239e5e2e79fda2a353d2a29122c1dc"
server_key_iv: "f73980e70995614e9fb166613617942a"
server_key_version: "encryptionkey@0"
I20260807 11:30:49.817930 3420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:49.835898 3420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:49.836771 3420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:49.837070 3420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:49.838038 3420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:49.839654 3420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:49.839747 3420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:49.839833 3420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:49.839895 3420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:49.874125 3420 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:37277
I20260807 11:30:49.874178 3563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:37277 every 8 connection(s)
I20260807 11:30:49.875538 3420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:30:49.884064 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 3420
I20260807 11:30:49.884280 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:30:49.884735 24989 external_mini_cluster.cc:1428] Setting key 8909b4740453d7888979f888bb08ebf6
I20260807 11:30:49.887760 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38327
--builtin_ntp_servers=127.24.103.84:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 11:30:49.888870 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:49.878231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:42369 (local address 127.24.103.126:38327)
0807 11:30:49.878418 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:49.878423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:49.879886 (+ 1463us) server_negotiation.cc:407] Connection header received
0807 11:30:49.881702 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:49.881708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:49.881807 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:49.881921 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:49.882994 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:49.883907 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:49.885872 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:49.886297 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:49.886396 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:49.888048 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:49.888053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:49.888056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:49.888074 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:49.888105 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:49.888117 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:49.888219 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:49.888444 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 11:30:49.888546 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:30:49.889894 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:49.878670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38327 (local address 127.24.103.66:42369)
0807 11:30:49.879714 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:49.879751 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:49.881410 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:49.882172 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:49.882180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:49.882264 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:49.882824 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:49.882839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:49.884077 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:49.884081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:49.885663 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:49.885676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:49.886548 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:49.886555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:49.886870 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:49.887672 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:30:49.887675 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:49.887714 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:30:49.887790 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:49.888219 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:49.888232 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:49.888242 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 11:30:49.888432 (+ 190us) client_negotiation.cc:240] Negotiation successful
0807 11:30:49.888830 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":405,"threads_started":1}
I20260807 11:30:49.891592 3564 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38327
I20260807 11:30:49.891914 3564 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:49.892891 3564 heartbeater.cc:511] Master 127.24.103.126:38327 requested a full tablet report, sending...
I20260807 11:30:49.894624 3206 ts_manager.cc:194] Registered new tserver with Master: 592de1aad6604fa689e68de4f892dcb8 (127.24.103.66:37277)
I20260807 11:30:49.895746 3206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.66:42369
W20260807 11:30:50.243502 3567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:50.243852 3567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:50.244050 3567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:50.252050 3567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:50.252153 3567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:50.252321 3567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:30:50.261806 3567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:50.263257 3567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:50.265116 3567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:50.278872 3576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:50.278851 3579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:50.279443 3575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:50.279897 3567 server_base.cc:1034] running on GCE node
I20260807 11:30:50.280467 3567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:50.281427 3567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:50.282635 3567 hybrid_clock.cc:648] HybridClock initialized: now 1786102250282583 us; error 39 us; skew 500 ppm
I20260807 11:30:50.282915 3567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:30:50.285467 3567 webserver.cc:460] Webserver started at http://127.24.103.67:42371/ using document root <none> and password file <none>
I20260807 11:30:50.286188 3567 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:50.286278 3567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:50.286551 3567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:50.289570 3567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "67a78ebb5b0445d6b4fc13ddf900335b"
format_stamp: "Formatted at 2026-08-07 11:30:50 on dist-test-slave-wl2h"
server_key: "e8123547a542bfa2b9c0f4f8d9a534bd"
server_key_iv: "9b8b285b0b9d17f76663af99baffb241"
server_key_version: "encryptionkey@0"
I20260807 11:30:50.290334 3567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "67a78ebb5b0445d6b4fc13ddf900335b"
format_stamp: "Formatted at 2026-08-07 11:30:50 on dist-test-slave-wl2h"
server_key: "e8123547a542bfa2b9c0f4f8d9a534bd"
server_key_iv: "9b8b285b0b9d17f76663af99baffb241"
server_key_version: "encryptionkey@0"
I20260807 11:30:50.295737 3567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 11:30:50.299430 3587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:50.300841 3567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260807 11:30:50.300992 3567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "67a78ebb5b0445d6b4fc13ddf900335b"
format_stamp: "Formatted at 2026-08-07 11:30:50 on dist-test-slave-wl2h"
server_key: "e8123547a542bfa2b9c0f4f8d9a534bd"
server_key_iv: "9b8b285b0b9d17f76663af99baffb241"
server_key_version: "encryptionkey@0"
I20260807 11:30:50.301201 3567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:50.314227 3567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:50.315228 3567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:50.315567 3567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:50.316545 3567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:50.318090 3567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:50.318226 3567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:50.318356 3567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:50.318454 3567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:50.354799 3567 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:41527
I20260807 11:30:50.354853 3713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:41527 every 8 connection(s)
I20260807 11:30:50.356181 3567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:30:50.359622 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 3567
I20260807 11:30:50.359841 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:30:50.360587 24989 external_mini_cluster.cc:1428] Setting key c2381f6d8f68958893eaded2f38f1e97
I20260807 11:30:50.370751 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:50.358909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:34921 (local address 127.24.103.126:38327)
0807 11:30:50.359155 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:50.359162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:50.362252 (+ 3090us) server_negotiation.cc:407] Connection header received
0807 11:30:50.364129 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:50.364135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:50.364240 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:50.364356 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:50.365407 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:50.366164 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:50.367780 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:50.368186 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:50.368327 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:50.369871 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:50.369876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:50.369880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:50.369899 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:50.369933 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:50.369946 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:50.370059 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:50.370288 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 11:30:50.370391 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:30:50.371810 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:50.359678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38327 (local address 127.24.103.67:34921)
0807 11:30:50.362086 (+ 2408us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:50.362131 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:50.363862 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:50.364586 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:50.364595 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:50.364693 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:50.365235 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:50.365253 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:50.366334 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:50.366338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:50.367629 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:50.367641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:50.368413 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:50.368421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:50.368665 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:50.369510 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:30:50.369514 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:50.369569 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:30:50.369646 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:50.370065 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:50.370078 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:50.370087 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 11:30:50.370280 (+ 193us) client_negotiation.cc:240] Negotiation successful
0807 11:30:50.370661 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1856,"thread_start_us":1624,"threads_started":1}
I20260807 11:30:50.373209 3714 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38327
I20260807 11:30:50.373519 3714 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:50.374336 3714 heartbeater.cc:511] Master 127.24.103.126:38327 requested a full tablet report, sending...
I20260807 11:30:50.376096 3206 ts_manager.cc:194] Registered new tserver with Master: 67a78ebb5b0445d6b4fc13ddf900335b (127.24.103.67:41527)
I20260807 11:30:50.376941 3206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.67:34921
I20260807 11:30:50.389122 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 11:30:50.396221 3418 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:50.394358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.24.103.126:38327)
0807 11:30:50.394663 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:50.394674 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:50.395115 (+ 441us) server_negotiation.cc:407] Connection header received
0807 11:30:50.395338 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:50.395343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:50.395412 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:50.395499 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:50.395974 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52720: BlockingRecv error: recv got EOF from 127.0.0.1:52720 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:30:50.397128 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 3272
I20260807 11:30:50.405822 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:50.398462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.24.103.65:35057)
0807 11:30:50.399068 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:50.399080 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:50.399118 (+ 38us) server_negotiation.cc:407] Connection header received
0807 11:30:50.399186 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:50.399192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:50.399389 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:50.399556 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:50.400000 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:50.401072 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:50.402406 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:50.402951 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:50.403164 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:50.404291 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:50.404300 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:50.404311 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:50.404904 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:50.404960 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:50.404980 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:50.405104 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:50.405282 (+ 178us) server_negotiation.cc:299] Negotiation successful
0807 11:30:50.405421 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":289,"threads_started":1}
I20260807 11:30:50.407116 3409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:50.490192 3416 heartbeater.cc:503] Master 127.24.103.126:38327 was elected leader, sending a full tablet report...
I20260807 11:30:50.491410 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 3272
W20260807 11:30:50.510519 3186 connection.cc:535] server connection from 127.24.103.65:34565 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 11:30:50.510913 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 3420
I20260807 11:30:50.519799 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:50.511915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.24.103.66:37277)
0807 11:30:50.512539 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:50.512554 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:50.512584 (+ 30us) server_negotiation.cc:407] Connection header received
0807 11:30:50.512660 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:50.512667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:50.512880 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:50.513013 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:50.513460 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:50.514502 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:50.516091 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:50.516672 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:50.516780 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:50.518084 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:50.518093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:50.518107 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:50.518763 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:50.518821 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:50.518841 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:50.518938 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:50.519231 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 11:30:50.519411 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":356,"threads_started":1}
I20260807 11:30:50.521263 3559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:50.605540 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 3420
I20260807 11:30:50.626273 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 3567
I20260807 11:30:50.635358 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:50.627442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.24.103.67:41527)
0807 11:30:50.628095 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:50.628102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:50.628133 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:30:50.628226 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:50.628233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:50.628457 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:50.628588 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:50.629000 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:50.630083 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:50.631860 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:50.632492 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:50.632597 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:50.633760 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:50.633769 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:50.633781 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:50.634451 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:50.634499 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:50.634524 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:50.634619 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:50.634787 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 11:30:50.634913 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":353,"threads_started":1}
I20260807 11:30:50.636729 3709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:50.720546 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 3567
I20260807 11:30:50.739530 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 3167
I20260807 11:30:50.740845 3240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:50.851163 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 3167
2026-08-07T11:30:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2603 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T11:30:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:30:50Z Disabled control of system clock
I20260807 11:30:50.944286 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34831
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:34831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 11:30:51.308107 3737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:51.308436 3737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:51.308529 3737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:51.316867 3737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:51.317008 3737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:51.317049 3737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:51.317099 3737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:51.317142 3737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:30:51.326994 3737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:34831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:51.328382 3737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:51.330214 3737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:51.342293 3745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:51.342448 3742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:51.342406 3743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:51.343303 3737 server_base.cc:1034] running on GCE node
I20260807 11:30:51.344125 3737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:51.345634 3737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:51.346890 3737 hybrid_clock.cc:648] HybridClock initialized: now 1786102251346841 us; error 42 us; skew 500 ppm
I20260807 11:30:51.347308 3737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:30:51.349751 3737 webserver.cc:460] Webserver started at http://127.24.103.126:43575/ using document root <none> and password file <none>
I20260807 11:30:51.350504 3737 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:51.350642 3737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:51.350989 3737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:51.354166 3737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "967f0904d1eb4b34bef42b9e3f9e4658"
format_stamp: "Formatted at 2026-08-07 11:30:51 on dist-test-slave-wl2h"
server_key: "73c890897d823800c7feffa14294d75d"
server_key_iv: "ce611ef560b2ccb856027e68d7b54dcd"
server_key_version: "encryptionkey@0"
I20260807 11:30:51.355091 3737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "967f0904d1eb4b34bef42b9e3f9e4658"
format_stamp: "Formatted at 2026-08-07 11:30:51 on dist-test-slave-wl2h"
server_key: "73c890897d823800c7feffa14294d75d"
server_key_iv: "ce611ef560b2ccb856027e68d7b54dcd"
server_key_version: "encryptionkey@0"
I20260807 11:30:51.360724 3737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:30:51.364490 3751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:51.366024 3737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:30:51.366240 3737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "967f0904d1eb4b34bef42b9e3f9e4658"
format_stamp: "Formatted at 2026-08-07 11:30:51 on dist-test-slave-wl2h"
server_key: "73c890897d823800c7feffa14294d75d"
server_key_iv: "ce611ef560b2ccb856027e68d7b54dcd"
server_key_version: "encryptionkey@0"
I20260807 11:30:51.366478 3737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:51.372350 3737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:51.373158 3737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:51.373474 3737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:51.397161 3737 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:34831
I20260807 11:30:51.397255 3812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:34831 every 8 connection(s)
I20260807 11:30:51.398629 3737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:30:51.402562 3813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:51.407385 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 3737
I20260807 11:30:51.407613 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:30:51.408469 24989 external_mini_cluster.cc:1428] Setting key 59e2baa357a8122aedd4d58b68befd77
I20260807 11:30:51.415192 3813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658: Bootstrap starting.
I20260807 11:30:51.417261 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:51.410783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.24.103.126:34831)
0807 11:30:51.411901 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:51.411911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:51.411956 (+ 45us) server_negotiation.cc:407] Connection header received
0807 11:30:51.413691 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:51.413713 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:51.414115 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:51.414434 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:51.415676 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:51.415682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:51.415694 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:51.415709 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:51.415773 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:51.415789 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:51.415966 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:51.416076 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 11:30:51.416357 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":438,"threads_started":1}
I20260807 11:30:51.418819 3813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:51.419929 3813 log.cc:824] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:51.422614 3813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658: No bootstrap required, opened a new log
I20260807 11:30:51.429894 3813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f0904d1eb4b34bef42b9e3f9e4658" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34831 } }
I20260807 11:30:51.430327 3813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:51.430440 3813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967f0904d1eb4b34bef42b9e3f9e4658, State: Initialized, Role: FOLLOWER
I20260807 11:30:51.431149 3813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f0904d1eb4b34bef42b9e3f9e4658" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34831 } }
I20260807 11:30:51.431403 3813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:30:51.431550 3813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:30:51.431742 3813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:51.434082 3813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f0904d1eb4b34bef42b9e3f9e4658" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34831 } }
I20260807 11:30:51.434602 3813 leader_election.cc:302] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 967f0904d1eb4b34bef42b9e3f9e4658; no voters:
I20260807 11:30:51.435308 3813 leader_election.cc:288] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:30:51.435442 3818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:51.436236 3818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [term 1 LEADER]: Becoming Leader. State: Replica: 967f0904d1eb4b34bef42b9e3f9e4658, State: Running, Role: LEADER
I20260807 11:30:51.436874 3818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f0904d1eb4b34bef42b9e3f9e4658" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34831 } }
I20260807 11:30:51.437536 3813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:51.440138 3819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "967f0904d1eb4b34bef42b9e3f9e4658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f0904d1eb4b34bef42b9e3f9e4658" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34831 } } }
I20260807 11:30:51.440539 3819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:51.441056 3820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 967f0904d1eb4b34bef42b9e3f9e4658. Latest consensus state: current_term: 1 leader_uuid: "967f0904d1eb4b34bef42b9e3f9e4658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967f0904d1eb4b34bef42b9e3f9e4658" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34831 } } }
I20260807 11:30:51.441408 3820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658 [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:51.441485 3828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:30:51.448372 3828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:30:51.458415 3828 catalog_manager.cc:1347] Generated new cluster ID: 9edae7cfe8fe44fa92ebdd00768fd8bc
I20260807 11:30:51.458585 3828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:30:51.477999 3828 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:30:51.478569 3828 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:30:51.509030 3828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 967f0904d1eb4b34bef42b9e3f9e4658: Generated new TSK 0
I20260807 11:30:51.510166 3828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:30:51.528015 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34831
--builtin_ntp_servers=127.24.103.84:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 11:30:51.886806 3839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:51.887190 3839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:51.887305 3839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:51.895172 3839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:51.895304 3839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:51.895423 3839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:30:51.905504 3839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:51.906986 3839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:51.909049 3839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:51.923627 3845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:51.923744 3844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:51.923692 3847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:51.924695 3839 server_base.cc:1034] running on GCE node
I20260807 11:30:51.925314 3839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:51.926297 3839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:51.927510 3839 hybrid_clock.cc:648] HybridClock initialized: now 1786102251927466 us; error 30 us; skew 500 ppm
I20260807 11:30:51.927861 3839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:30:51.935470 3839 webserver.cc:460] Webserver started at http://127.24.103.65:41833/ using document root <none> and password file <none>
I20260807 11:30:51.936347 3839 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:51.936487 3839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:51.936899 3839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:51.940698 3839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "0a958e3f2fd746c2896e2fdb25d90907"
format_stamp: "Formatted at 2026-08-07 11:30:51 on dist-test-slave-wl2h"
server_key: "b2717f8df95830802884b3f2e0179ef3"
server_key_iv: "06bc1885594293fbafd63cde39723c60"
server_key_version: "encryptionkey@0"
I20260807 11:30:51.941759 3839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "0a958e3f2fd746c2896e2fdb25d90907"
format_stamp: "Formatted at 2026-08-07 11:30:51 on dist-test-slave-wl2h"
server_key: "b2717f8df95830802884b3f2e0179ef3"
server_key_iv: "06bc1885594293fbafd63cde39723c60"
server_key_version: "encryptionkey@0"
I20260807 11:30:51.947870 3839 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 11:30:51.951824 3854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:51.953346 3839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:30:51.953553 3839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "0a958e3f2fd746c2896e2fdb25d90907"
format_stamp: "Formatted at 2026-08-07 11:30:51 on dist-test-slave-wl2h"
server_key: "b2717f8df95830802884b3f2e0179ef3"
server_key_iv: "06bc1885594293fbafd63cde39723c60"
server_key_version: "encryptionkey@0"
I20260807 11:30:51.953827 3839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:51.960577 3839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:51.961474 3839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:51.961784 3839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:51.962834 3839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:51.964443 3839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:51.964571 3839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:51.964736 3839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:51.964841 3839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:52.005707 3839 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:36459
I20260807 11:30:52.005765 3985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:36459 every 8 connection(s)
I20260807 11:30:52.007227 3839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:30:52.011415 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 3839
I20260807 11:30:52.011663 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:30:52.012522 24989 external_mini_cluster.cc:1428] Setting key 985b55a7d3721aaa02ae99d8ca3db4d9
I20260807 11:30:52.015695 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34831
--builtin_ntp_servers=127.24.103.84:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 11:30:52.016846 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:52.010413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:34523 (local address 127.24.103.126:34831)
0807 11:30:52.010943 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:52.010952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:52.012329 (+ 1377us) server_negotiation.cc:407] Connection header received
0807 11:30:52.014109 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:52.014115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:52.014174 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:52.014303 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:52.015816 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:52.015822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:52.015827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:52.015851 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:52.015906 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:52.015923 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:52.016050 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:52.016339 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 11:30:52.016417 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":232,"threads_started":1}
I20260807 11:30:52.017581 3989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:52.010971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34831 (local address 127.24.103.65:34523)
0807 11:30:52.012126 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:52.012174 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:52.013840 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:52.014622 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:52.014635 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:52.014750 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:52.015500 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:30:52.015503 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:52.015542 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:30:52.015629 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:52.016105 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:52.016113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:52.016119 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 11:30:52.016348 (+ 229us) client_negotiation.cc:240] Negotiation successful
0807 11:30:52.016550 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":363,"threads_started":1}
I20260807 11:30:52.019325 3986 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34831
I20260807 11:30:52.019639 3986 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:52.020239 3986 heartbeater.cc:511] Master 127.24.103.126:34831 requested a full tablet report, sending...
I20260807 11:30:52.022588 3773 ts_manager.cc:194] Registered new tserver with Master: 0a958e3f2fd746c2896e2fdb25d90907 (127.24.103.65:36459)
W20260807 11:30:52.362838 3990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:52.363209 3990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:52.363309 3990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:52.370837 3990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:52.370935 3990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:52.371104 3990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:30:52.380681 3990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:52.381965 3990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:52.383833 3990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:52.397090 3995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:52.397150 3996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:52.397197 3998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:52.400609 3990 server_base.cc:1034] running on GCE node
I20260807 11:30:52.401324 3990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:52.402354 3990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:52.403656 3990 hybrid_clock.cc:648] HybridClock initialized: now 1786102252403596 us; error 41 us; skew 500 ppm
I20260807 11:30:52.404007 3990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:30:52.407277 3990 webserver.cc:460] Webserver started at http://127.24.103.66:38425/ using document root <none> and password file <none>
I20260807 11:30:52.408306 3990 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:52.408442 3990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:52.408922 3990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:52.413597 3990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "d84c7e4e296e44eca2c879cb9f044f46"
format_stamp: "Formatted at 2026-08-07 11:30:52 on dist-test-slave-wl2h"
server_key: "32e1da0948f09cc63c23229bb3244ee2"
server_key_iv: "a21d3953c93bfb09a1b4ed87738607da"
server_key_version: "encryptionkey@0"
I20260807 11:30:52.414834 3990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "d84c7e4e296e44eca2c879cb9f044f46"
format_stamp: "Formatted at 2026-08-07 11:30:52 on dist-test-slave-wl2h"
server_key: "32e1da0948f09cc63c23229bb3244ee2"
server_key_iv: "a21d3953c93bfb09a1b4ed87738607da"
server_key_version: "encryptionkey@0"
I20260807 11:30:52.422566 3990 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260807 11:30:52.427088 4004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:52.428578 3990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260807 11:30:52.428786 3990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "d84c7e4e296e44eca2c879cb9f044f46"
format_stamp: "Formatted at 2026-08-07 11:30:52 on dist-test-slave-wl2h"
server_key: "32e1da0948f09cc63c23229bb3244ee2"
server_key_iv: "a21d3953c93bfb09a1b4ed87738607da"
server_key_version: "encryptionkey@0"
I20260807 11:30:52.429042 3990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:52.435207 3990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:52.436044 3990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:52.436326 3990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:52.437364 3990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:52.439196 3990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:52.439327 3990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:52.439450 3990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:52.439541 3990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:52.476212 3990 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:45675
I20260807 11:30:52.476305 4132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:45675 every 8 connection(s)
I20260807 11:30:52.477582 3990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:30:52.478821 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 3990
I20260807 11:30:52.478992 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:30:52.479643 24989 external_mini_cluster.cc:1428] Setting key 18cbf02362dab6ec160908b1990e64c8
I20260807 11:30:52.482980 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34831
--builtin_ntp_servers=127.24.103.84:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 11:30:52.487354 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:52.480526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:39229 (local address 127.24.103.126:34831)
0807 11:30:52.480735 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:52.480740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:52.482622 (+ 1882us) server_negotiation.cc:407] Connection header received
0807 11:30:52.484385 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:52.484392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:52.484464 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:52.484748 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:52.486511 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:52.486517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:52.486522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:52.486544 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:52.486590 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:52.486603 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:52.486770 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:52.486956 (+ 186us) server_negotiation.cc:299] Negotiation successful
0807 11:30:52.487063 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:30:52.488338 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:52.481093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34831 (local address 127.24.103.66:39229)
0807 11:30:52.482455 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:52.482490 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:52.484116 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:52.485059 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:52.485143 (+ 84us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:52.485265 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:52.486151 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:30:52.486153 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:52.486193 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:30:52.486313 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:52.486705 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:52.486715 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:52.486723 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 11:30:52.487065 (+ 342us) client_negotiation.cc:240] Negotiation successful
0807 11:30:52.487344 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":670,"threads_started":1}
I20260807 11:30:52.489938 4133 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34831
I20260807 11:30:52.490254 4133 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:52.490876 4133 heartbeater.cc:511] Master 127.24.103.126:34831 requested a full tablet report, sending...
I20260807 11:30:52.492394 3773 ts_manager.cc:194] Registered new tserver with Master: d84c7e4e296e44eca2c879cb9f044f46 (127.24.103.66:45675)
W20260807 11:30:52.848125 4137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:52.848459 4137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:52.848631 4137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:52.856144 4137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:52.856294 4137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:52.856477 4137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:30:52.866039 4137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:34831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:52.867436 4137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:52.869325 4137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:52.883697 4145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:52.883651 4143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:52.883800 4142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:52.884667 4137 server_base.cc:1034] running on GCE node
I20260807 11:30:52.885347 4137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:52.886442 4137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:52.887710 4137 hybrid_clock.cc:648] HybridClock initialized: now 1786102252887641 us; error 75 us; skew 500 ppm
I20260807 11:30:52.888113 4137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:30:52.890534 4137 webserver.cc:460] Webserver started at http://127.24.103.67:42715/ using document root <none> and password file <none>
I20260807 11:30:52.891390 4137 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:52.891530 4137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:52.891867 4137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:52.894920 4137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "80b59896642248cb8b9fbb8bcf824699"
format_stamp: "Formatted at 2026-08-07 11:30:52 on dist-test-slave-wl2h"
server_key: "0ce0e2db0c6f7b30122fc2584d728f17"
server_key_iv: "70cb749effdf19a2cf12e76ef1ba6d00"
server_key_version: "encryptionkey@0"
I20260807 11:30:52.895828 4137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "80b59896642248cb8b9fbb8bcf824699"
format_stamp: "Formatted at 2026-08-07 11:30:52 on dist-test-slave-wl2h"
server_key: "0ce0e2db0c6f7b30122fc2584d728f17"
server_key_iv: "70cb749effdf19a2cf12e76ef1ba6d00"
server_key_version: "encryptionkey@0"
I20260807 11:30:52.901340 4137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:30:52.905005 4151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:52.906507 4137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 11:30:52.906709 4137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "80b59896642248cb8b9fbb8bcf824699"
format_stamp: "Formatted at 2026-08-07 11:30:52 on dist-test-slave-wl2h"
server_key: "0ce0e2db0c6f7b30122fc2584d728f17"
server_key_iv: "70cb749effdf19a2cf12e76ef1ba6d00"
server_key_version: "encryptionkey@0"
I20260807 11:30:52.907060 4137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:52.913147 4137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:52.913959 4137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:52.914247 4137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:52.915274 4137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:52.916724 4137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:52.916841 4137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:52.916971 4137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:52.917064 4137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:52.952411 4137 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:33863
I20260807 11:30:52.952499 4278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:33863 every 8 connection(s)
I20260807 11:30:52.953820 4137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:30:52.954882 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 4137
I20260807 11:30:52.955088 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:30:52.955644 24989 external_mini_cluster.cc:1428] Setting key 26cac8f12645511a3805e8726758a53d
I20260807 11:30:52.962976 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:52.956902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:54293 (local address 127.24.103.126:34831)
0807 11:30:52.957136 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:52.957143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:52.958733 (+ 1590us) server_negotiation.cc:407] Connection header received
0807 11:30:52.960621 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:52.960627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:52.960695 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:52.960816 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:52.962149 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:52.962155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:52.962160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:52.962179 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:52.962234 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:52.962246 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:52.962375 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:52.962611 (+ 236us) server_negotiation.cc:299] Negotiation successful
0807 11:30:52.962681 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:30:52.963912 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:52.957485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:34831 (local address 127.24.103.67:54293)
0807 11:30:52.958585 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:52.958628 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:52.960372 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:52.961054 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:52.961062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:52.961148 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:52.961866 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:30:52.961869 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:52.961905 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:30:52.961987 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:52.962351 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:52.962361 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:52.962368 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 11:30:52.962679 (+ 311us) client_negotiation.cc:240] Negotiation successful
0807 11:30:52.962930 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":360,"threads_started":1}
I20260807 11:30:52.965262 4279 heartbeater.cc:347] Connected to a master server at 127.24.103.126:34831
I20260807 11:30:52.965553 4279 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:52.966135 4279 heartbeater.cc:511] Master 127.24.103.126:34831 requested a full tablet report, sending...
I20260807 11:30:52.967528 3773 ts_manager.cc:194] Registered new tserver with Master: 80b59896642248cb8b9fbb8bcf824699 (127.24.103.67:33863)
I20260807 11:30:52.971580 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 11:30:52.978367 3988 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:30:52.976553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.24.103.126:34831)
0807 11:30:52.976738 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:52.976743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:52.977256 (+ 513us) server_negotiation.cc:407] Connection header received
0807 11:30:52.977376 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:52.977380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:52.977439 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:52.977525 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:52.977978 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56712: BlockingRecv error: recv got EOF from 127.0.0.1:56712 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:30:52.979588 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 3839
I20260807 11:30:52.983561 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:52.980747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.24.103.65:36459)
0807 11:30:52.981332 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:52.981340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:52.981367 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:30:52.981427 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:52.981432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:52.981643 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:52.981770 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:52.982689 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:52.982695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:52.982706 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:52.982717 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:52.982759 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:52.982773 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:52.982887 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:52.983181 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 11:30:52.983274 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":292,"threads_started":1}
I20260807 11:30:52.984920 3981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:53.067719 3986 heartbeater.cc:503] Master 127.24.103.126:34831 was elected leader, sending a full tablet report...
I20260807 11:30:53.069144 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 3839
W20260807 11:30:53.090579 3755 connection.cc:535] server connection from 127.24.103.65:34523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 11:30:53.090973 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 3990
I20260807 11:30:53.095465 4291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:53.092286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.24.103.66:45675)
0807 11:30:53.092892 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:53.092900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:53.092929 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:30:53.093002 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:53.093009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:53.093261 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:53.093485 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:53.094634 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:53.094641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:53.094653 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:53.094667 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:53.094709 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:53.094724 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:53.094920 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:53.095001 (+ 81us) server_negotiation.cc:299] Negotiation successful
0807 11:30:53.095145 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":319,"threads_started":1}
I20260807 11:30:53.097018 4128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:53.183691 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 3990
I20260807 11:30:53.205220 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 4137
I20260807 11:30:53.211416 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:53.206240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.24.103.67:33863)
0807 11:30:53.207934 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:53.207942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:53.207978 (+ 36us) server_negotiation.cc:407] Connection header received
0807 11:30:53.208079 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:53.208087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:53.208315 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:53.208451 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:53.210267 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:53.210275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:53.210287 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:30:53.210304 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:53.210363 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:30:53.210383 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:53.210606 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:53.210717 (+ 111us) server_negotiation.cc:299] Negotiation successful
0807 11:30:53.210844 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":444,"threads_started":1}
I20260807 11:30:53.212836 4274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:53.300022 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 4137
I20260807 11:30:53.319867 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 3737
I20260807 11:30:53.321254 3808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:53.433295 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 3737
2026-08-07T11:30:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2583 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:30:53 dist-test-slave-wl2h krb5kdc[4298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:30:53 dist-test-slave-wl2h krb5kdc[4298](info): set up 2 sockets
Aug 07 11:30:53 dist-test-slave-wl2h krb5kdc[4298](info): commencing operation
krb5kdc: starting...
W20260807 11:30:54.655546 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:30:54 dist-test-slave-wl2h krb5kdc[4298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:30:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:30:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:30:54.984100 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:45093
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:45093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:55.360422 4311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:55.360735 4311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:55.360831 4311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:55.368225 4311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:30:55.368340 4311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:55.368387 4311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:55.368436 4311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:30:55.368480 4311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:30:55.381263 4311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:45093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:45093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:55.382997 4311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:55.384409 4311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:55.397437 4318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:55.397607 4319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:55.397570 4322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:55.398607 4311 server_base.cc:1034] running on GCE node
I20260807 11:30:55.399555 4311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:55.401142 4311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:55.402371 4311 hybrid_clock.cc:648] HybridClock initialized: now 1786102255402298 us; error 55 us; skew 500 ppm
I20260807 11:30:55.402773 4311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:55 dist-test-slave-wl2h krb5kdc[4298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102255, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:55.406505 4311 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:30:55.408378 4311 webserver.cc:460] Webserver started at http://127.24.103.126:44623/ using document root <none> and password file <none>
I20260807 11:30:55.409178 4311 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:55.409354 4311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:55.409725 4311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:55.413350 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "4da0d4c46f1a4341b50b91dbb47f167a"
format_stamp: "Formatted at 2026-08-07 11:30:55 on dist-test-slave-wl2h"
server_key: "93f8cb4e39bf6c5f77651b59c1144a0c"
server_key_iv: "e52f2f067cbb3607f4e95af8531e48ac"
server_key_version: "encryptionkey@0"
I20260807 11:30:55.414261 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "4da0d4c46f1a4341b50b91dbb47f167a"
format_stamp: "Formatted at 2026-08-07 11:30:55 on dist-test-slave-wl2h"
server_key: "93f8cb4e39bf6c5f77651b59c1144a0c"
server_key_iv: "e52f2f067cbb3607f4e95af8531e48ac"
server_key_version: "encryptionkey@0"
I20260807 11:30:55.420383 4311 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 11:30:55.424887 4329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:55.426370 4311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:30:55.426589 4311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "4da0d4c46f1a4341b50b91dbb47f167a"
format_stamp: "Formatted at 2026-08-07 11:30:55 on dist-test-slave-wl2h"
server_key: "93f8cb4e39bf6c5f77651b59c1144a0c"
server_key_iv: "e52f2f067cbb3607f4e95af8531e48ac"
server_key_version: "encryptionkey@0"
I20260807 11:30:55.426817 4311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:55.452543 4311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:55.457403 4311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:55.457741 4311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:55.478570 4311 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:45093
I20260807 11:30:55.478636 4389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:45093 every 8 connection(s)
I20260807 11:30:55.480139 4311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:30:55.484194 4390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:55.487927 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 4311
I20260807 11:30:55.488157 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:30:55.488852 24989 external_mini_cluster.cc:1428] Setting key b9d2e164139546755d4f3173eb3e6026
I20260807 11:30:55.497052 4390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a: Bootstrap starting.
I20260807 11:30:55.501128 4390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:55.502359 4390 log.cc:824] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:55.505481 4390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a: No bootstrap required, opened a new log
Aug 07 11:30:55 dist-test-slave-wl2h krb5kdc[4298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:55.516170 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:55.491467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.24.103.126:45093)
0807 11:30:55.492326 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:55.492347 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:55.492398 (+ 51us) server_negotiation.cc:407] Connection header received
0807 11:30:55.495098 (+ 2700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:55.495128 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:55.495641 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:55.496211 (+ 570us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:55.497841 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:55.499904 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:55.501318 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:55.502041 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:55.502257 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:55.507178 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:55.507187 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:55.507203 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:55.507251 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:55.510780 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:55.511899 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:55.511907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:55.511914 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:55.512127 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:55.512700 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:55.512704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:55.512707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:55.513525 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:55.513799 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:55.514410 (+ 611us) server_negotiation.cc:299] Negotiation successful
0807 11:30:55.514951 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":370,"threads_started":1}
I20260807 11:30:55.516041 4390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da0d4c46f1a4341b50b91dbb47f167a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45093 } }
I20260807 11:30:55.516579 4390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:55.516727 4390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da0d4c46f1a4341b50b91dbb47f167a, State: Initialized, Role: FOLLOWER
I20260807 11:30:55.517443 4390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da0d4c46f1a4341b50b91dbb47f167a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45093 } }
I20260807 11:30:55.517742 4390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:30:55.517932 4390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:30:55.518265 4390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:55.521265 4390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da0d4c46f1a4341b50b91dbb47f167a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45093 } }
I20260807 11:30:55.521944 4390 leader_election.cc:302] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4da0d4c46f1a4341b50b91dbb47f167a; no voters:
I20260807 11:30:55.522727 4390 leader_election.cc:288] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:30:55.522929 4395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:55.523904 4395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [term 1 LEADER]: Becoming Leader. State: Replica: 4da0d4c46f1a4341b50b91dbb47f167a, State: Running, Role: LEADER
I20260807 11:30:55.524526 4395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da0d4c46f1a4341b50b91dbb47f167a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45093 } }
I20260807 11:30:55.525174 4390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:30:55.528119 4397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4da0d4c46f1a4341b50b91dbb47f167a. Latest consensus state: current_term: 1 leader_uuid: "4da0d4c46f1a4341b50b91dbb47f167a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da0d4c46f1a4341b50b91dbb47f167a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45093 } } }
I20260807 11:30:55.528620 4397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:55.529279 4396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4da0d4c46f1a4341b50b91dbb47f167a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da0d4c46f1a4341b50b91dbb47f167a" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45093 } } }
I20260807 11:30:55.529693 4396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a [sys.catalog]: This master's current role is: LEADER
I20260807 11:30:55.531836 4404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:30:55.537811 4404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:30:55.548775 4404 catalog_manager.cc:1347] Generated new cluster ID: 9b0cb749f9ea4615bff1bf12a0e21fb2
I20260807 11:30:55.548933 4404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:30:55.569795 4404 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:30:55.571245 4404 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:30:55.581370 4404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a: Generated new TSK 0
I20260807 11:30:55.582321 4404 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:30:55.669095 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45093
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:56.027550 4418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:56.027880 4418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:56.028047 4418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:56.036065 4418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:56.036193 4418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:56.036406 4418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:30:56.046727 4418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:56.048214 4418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:56.049603 4418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:56.063925 4424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:56.063946 4425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:56.064493 4428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:56.064924 4418 server_base.cc:1034] running on GCE node
I20260807 11:30:56.065500 4418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:56.066396 4418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:56.067664 4418 hybrid_clock.cc:648] HybridClock initialized: now 1786102256067592 us; error 72 us; skew 500 ppm
I20260807 11:30:56.067997 4418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:56 dist-test-slave-wl2h krb5kdc[4298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:56.071466 4418 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:30:56.073058 4418 webserver.cc:460] Webserver started at http://127.24.103.65:39693/ using document root <none> and password file <none>
I20260807 11:30:56.073750 4418 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:56.073855 4418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:56.074112 4418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:56.077175 4418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "3d77b4c591c74946a97c18cac9ce12ec"
format_stamp: "Formatted at 2026-08-07 11:30:56 on dist-test-slave-wl2h"
server_key: "7196b20d11f300863f496b9055b75eb8"
server_key_iv: "539b4cf4369bedfe19358aef030e00bb"
server_key_version: "encryptionkey@0"
I20260807 11:30:56.077896 4418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "3d77b4c591c74946a97c18cac9ce12ec"
format_stamp: "Formatted at 2026-08-07 11:30:56 on dist-test-slave-wl2h"
server_key: "7196b20d11f300863f496b9055b75eb8"
server_key_iv: "539b4cf4369bedfe19358aef030e00bb"
server_key_version: "encryptionkey@0"
I20260807 11:30:56.083541 4418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:30:56.087267 4435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:56.088852 4418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:30:56.089003 4418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "3d77b4c591c74946a97c18cac9ce12ec"
format_stamp: "Formatted at 2026-08-07 11:30:56 on dist-test-slave-wl2h"
server_key: "7196b20d11f300863f496b9055b75eb8"
server_key_iv: "539b4cf4369bedfe19358aef030e00bb"
server_key_version: "encryptionkey@0"
I20260807 11:30:56.089188 4418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:56.116029 4418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:56.121243 4418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:56.121600 4418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:56.122665 4418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:56.124161 4418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:56.124241 4418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:56.124433 4418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:56.124487 4418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:56.163230 4418 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:41223
I20260807 11:30:56.163338 4562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:41223 every 8 connection(s)
I20260807 11:30:56.164773 4418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:30:56.171563 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 4418
I20260807 11:30:56.171787 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:30:56.172683 24989 external_mini_cluster.cc:1428] Setting key 5bbc98273bd92aac156341ba7f9d7492
Aug 07 11:30:56 dist-test-slave-wl2h krb5kdc[4298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:56.188442 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:56.167659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:58047 (local address 127.24.103.126:45093)
0807 11:30:56.168199 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:56.168203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:56.169039 (+ 836us) server_negotiation.cc:407] Connection header received
0807 11:30:56.171077 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:56.171082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:56.171192 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:56.171319 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:56.173047 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:56.174002 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:56.175917 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:56.176329 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:56.176478 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:56.180453 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:56.180463 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:56.180470 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:56.180519 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:56.184431 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:56.185522 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:56.185530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:56.185536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:56.185703 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:56.186203 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:56.186210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:56.186215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:56.186610 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:56.186769 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:56.187687 (+ 918us) server_negotiation.cc:299] Negotiation successful
0807 11:30:56.187975 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":306,"threads_started":1}
I20260807 11:30:56.189320 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:56.168121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45093 (local address 127.24.103.65:58047)
0807 11:30:56.168873 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:56.168913 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:56.170755 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:56.171569 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:56.171578 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:56.172216 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:56.172859 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:56.172879 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:56.174187 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:56.174192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:56.175751 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:56.175764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:56.176585 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:56.176592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:56.176847 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:56.177578 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:56.177601 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:56.180153 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:56.184634 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:56.184646 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:56.184675 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:56.185377 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:56.185826 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:56.185833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:56.185839 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:56.186081 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:56.186741 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:56.186752 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:56.187230 (+ 478us) client_negotiation.cc:769] Sending connection context
0807 11:30:56.187516 (+ 286us) client_negotiation.cc:240] Negotiation successful
0807 11:30:56.187938 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":289,"threads_started":1}
I20260807 11:30:56.191306 4563 heartbeater.cc:347] Connected to a master server at 127.24.103.126:45093
I20260807 11:30:56.191679 4563 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:56.192617 4563 heartbeater.cc:511] Master 127.24.103.126:45093 requested a full tablet report, sending...
I20260807 11:30:56.195196 4348 ts_manager.cc:194] Registered new tserver with Master: 3d77b4c591c74946a97c18cac9ce12ec (127.24.103.65:41223)
I20260807 11:30:56.196836 4348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:58047
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:30:56.248158 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45093
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:56.612296 4569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:56.612598 4569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:56.612696 4569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:56.620106 4569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:56.620240 4569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:56.620361 4569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:30:56.630642 4569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:56.632336 4569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:56.633781 4569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:56.647679 4575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:56.647716 4576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:56.648212 4578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:56.649031 4569 server_base.cc:1034] running on GCE node
I20260807 11:30:56.649740 4569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:56.650797 4569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:56.652071 4569 hybrid_clock.cc:648] HybridClock initialized: now 1786102256652020 us; error 59 us; skew 500 ppm
I20260807 11:30:56.652444 4569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:56 dist-test-slave-wl2h krb5kdc[4298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:56.656075 4569 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:30:56.657866 4569 webserver.cc:460] Webserver started at http://127.24.103.66:34623/ using document root <none> and password file <none>
I20260807 11:30:56.658643 4569 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:56.658807 4569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:56.659247 4569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:56.662549 4569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "497c3d00c785408088d524bf038d52b0"
format_stamp: "Formatted at 2026-08-07 11:30:56 on dist-test-slave-wl2h"
server_key: "a73832602829d71111f3222cc7bad5f9"
server_key_iv: "06bbbe3767a5e803dad287f4f53bd0f3"
server_key_version: "encryptionkey@0"
I20260807 11:30:56.663486 4569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "497c3d00c785408088d524bf038d52b0"
format_stamp: "Formatted at 2026-08-07 11:30:56 on dist-test-slave-wl2h"
server_key: "a73832602829d71111f3222cc7bad5f9"
server_key_iv: "06bbbe3767a5e803dad287f4f53bd0f3"
server_key_version: "encryptionkey@0"
I20260807 11:30:56.669313 4569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:30:56.673305 4585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:56.675094 4569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:30:56.675288 4569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "497c3d00c785408088d524bf038d52b0"
format_stamp: "Formatted at 2026-08-07 11:30:56 on dist-test-slave-wl2h"
server_key: "a73832602829d71111f3222cc7bad5f9"
server_key_iv: "06bbbe3767a5e803dad287f4f53bd0f3"
server_key_version: "encryptionkey@0"
I20260807 11:30:56.675514 4569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:56.696290 4569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:56.701413 4569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:56.701777 4569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:56.702958 4569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:56.704851 4569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:56.704933 4569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:56.705154 4569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:56.705214 4569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:56.742753 4569 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:46293
I20260807 11:30:56.742810 4711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:46293 every 8 connection(s)
I20260807 11:30:56.744149 4569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:30:56.751792 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 4569
I20260807 11:30:56.752007 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:30:56.752483 24989 external_mini_cluster.cc:1428] Setting key 8d12184a0203fd3b3bd90806ed90ffd3
Aug 07 11:30:56 dist-test-slave-wl2h krb5kdc[4298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:56.766197 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:56.746859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:52443 (local address 127.24.103.126:45093)
0807 11:30:56.747347 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:56.747351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:56.748282 (+ 931us) server_negotiation.cc:407] Connection header received
0807 11:30:56.750153 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:56.750159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:56.750265 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:56.750392 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:56.752110 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:56.752900 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:56.755238 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:56.755706 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:56.755852 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:56.759784 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:56.759789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:56.759793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:56.759831 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:56.762682 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:56.763603 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:56.763607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:56.763610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:56.763747 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:56.764291 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:56.764301 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:56.764303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:56.764591 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:56.764763 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:56.765534 (+ 771us) server_negotiation.cc:299] Negotiation successful
0807 11:30:56.765800 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":283,"threads_started":1}
I20260807 11:30:56.767345 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:56.747343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45093 (local address 127.24.103.66:52443)
0807 11:30:56.748105 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:56.748159 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:56.749873 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:56.750641 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:56.750649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:56.751274 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:56.751898 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:56.751925 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:56.753087 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:56.753092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:56.755000 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:56.755087 (+ 87us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:56.755973 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:56.755983 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:56.756238 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:56.757026 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:56.757057 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:56.759492 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:56.762854 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:56.762864 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:56.762884 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:56.763440 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:56.763875 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:56.763880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:56.763884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:56.764151 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:56.764732 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:56.764741 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:56.765134 (+ 393us) client_negotiation.cc:769] Sending connection context
0807 11:30:56.765424 (+ 290us) client_negotiation.cc:240] Negotiation successful
0807 11:30:56.766005 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":290,"threads_started":1}
I20260807 11:30:56.769052 4712 heartbeater.cc:347] Connected to a master server at 127.24.103.126:45093
I20260807 11:30:56.769434 4712 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:56.770267 4712 heartbeater.cc:511] Master 127.24.103.126:45093 requested a full tablet report, sending...
I20260807 11:30:56.772082 4348 ts_manager.cc:194] Registered new tserver with Master: 497c3d00c785408088d524bf038d52b0 (127.24.103.66:46293)
I20260807 11:30:56.772992 4348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:52443
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:30:56.824227 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45093
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:30:57.181725 4718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:30:57.182046 4718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:30:57.182267 4718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:30:57.190485 4718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:30:57.190623 4718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:30:57.190840 4718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:30:57.200976 4563 heartbeater.cc:503] Master 127.24.103.126:45093 was elected leader, sending a full tablet report...
I20260807 11:30:57.203505 4718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:30:57.204941 4718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:30:57.206369 4718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:30:57.221248 4723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:57.221359 4724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:30:57.221423 4726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:30:57.221861 4718 server_base.cc:1034] running on GCE node
I20260807 11:30:57.222707 4718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:30:57.223803 4718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:30:57.225060 4718 hybrid_clock.cc:648] HybridClock initialized: now 1786102257225007 us; error 41 us; skew 500 ppm
I20260807 11:30:57.225500 4718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:30:57 dist-test-slave-wl2h krb5kdc[4298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102257, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:30:57.233609 4718 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:30:57.235479 4718 webserver.cc:460] Webserver started at http://127.24.103.67:46721/ using document root <none> and password file <none>
I20260807 11:30:57.236332 4718 fs_manager.cc:359] Metadata directory not provided
I20260807 11:30:57.236485 4718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:30:57.236807 4718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:30:57.239990 4718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "79697639ad884c7f8ef5c57076be62f9"
format_stamp: "Formatted at 2026-08-07 11:30:57 on dist-test-slave-wl2h"
server_key: "e5dfe9e36088d4f8bedd95859585e68a"
server_key_iv: "2b83c09bbdcd7758ced674b06796d989"
server_key_version: "encryptionkey@0"
I20260807 11:30:57.240782 4718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "79697639ad884c7f8ef5c57076be62f9"
format_stamp: "Formatted at 2026-08-07 11:30:57 on dist-test-slave-wl2h"
server_key: "e5dfe9e36088d4f8bedd95859585e68a"
server_key_iv: "2b83c09bbdcd7758ced674b06796d989"
server_key_version: "encryptionkey@0"
I20260807 11:30:57.246857 4718 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20260807 11:30:57.250705 4733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:57.252313 4718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:30:57.252509 4718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "79697639ad884c7f8ef5c57076be62f9"
format_stamp: "Formatted at 2026-08-07 11:30:57 on dist-test-slave-wl2h"
server_key: "e5dfe9e36088d4f8bedd95859585e68a"
server_key_iv: "2b83c09bbdcd7758ced674b06796d989"
server_key_version: "encryptionkey@0"
I20260807 11:30:57.252766 4718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:30:57.269728 4718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:30:57.274356 4718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:30:57.274744 4718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:30:57.275887 4718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:30:57.277374 4718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:30:57.277477 4718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:57.277632 4718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:30:57.277709 4718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:30:57.315644 4718 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:39641
I20260807 11:30:57.315697 4858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:39641 every 8 connection(s)
I20260807 11:30:57.317116 4718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:30:57.318357 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 4718
I20260807 11:30:57.318559 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:30:57.319237 24989 external_mini_cluster.cc:1428] Setting key cff5c3c94aa2fed294f7bfafbfafcca0
Aug 07 11:30:57 dist-test-slave-wl2h krb5kdc[4298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102257, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:30:57.341631 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.320663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:41615 (local address 127.24.103.126:45093)
0807 11:30:57.321292 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:57.321300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:57.322712 (+ 1412us) server_negotiation.cc:407] Connection header received
0807 11:30:57.324916 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:57.324922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:57.325053 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:57.325212 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:57.327267 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.328175 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.330032 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.330565 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.330719 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.335113 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:57.335120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:57.335126 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:57.335175 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:57.338027 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:57.338879 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:57.338886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:57.338890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:57.339091 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:57.339601 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:57.339607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:57.339611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:57.339923 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:57.340112 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:57.340934 (+ 822us) server_negotiation.cc:299] Negotiation successful
0807 11:30:57.341191 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":277,"threads_started":1}
I20260807 11:30:57.342664 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.321350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45093 (local address 127.24.103.67:41615)
0807 11:30:57.322499 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:57.322565 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:57.324578 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:57.325591 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:57.325603 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:57.326262 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:30:57.326987 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.327079 (+ 92us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.328370 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.328378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.329841 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.329856 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.330830 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.330840 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.331199 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.331966 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:30:57.331997 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:30:57.334854 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:30:57.338201 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:57.338212 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:57.338231 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:57.338728 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:57.339234 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:30:57.339239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:30:57.339243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:30:57.339456 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:30:57.340078 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:30:57.340087 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:30:57.340478 (+ 391us) client_negotiation.cc:769] Sending connection context
0807 11:30:57.340813 (+ 335us) client_negotiation.cc:240] Negotiation successful
0807 11:30:57.341424 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":698,"threads_started":1}
I20260807 11:30:57.344363 4859 heartbeater.cc:347] Connected to a master server at 127.24.103.126:45093
I20260807 11:30:57.344669 4859 heartbeater.cc:464] Registering TS with master...
I20260807 11:30:57.345407 4859 heartbeater.cc:511] Master 127.24.103.126:45093 requested a full tablet report, sending...
I20260807 11:30:57.347720 4348 ts_manager.cc:194] Registered new tserver with Master: 79697639ad884c7f8ef5c57076be62f9 (127.24.103.67:39641)
I20260807 11:30:57.348660 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:30:57.349076 4348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:41615
I20260807 11:30:57.368955 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.354090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.24.103.126:45093)
0807 11:30:57.354343 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:57.354348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:57.354965 (+ 617us) server_negotiation.cc:407] Connection header received
0807 11:30:57.355196 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:57.355201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:57.355286 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:57.355374 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:57.357284 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.358141 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.359791 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.360234 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.360345 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.363372 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:57.363378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:57.363381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:57.363422 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:57.366066 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:57.366832 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:57.366836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:57.366839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:57.366979 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:57.367512 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:57.367516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:57.367519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:57.367861 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:57.368051 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:57.368482 (+ 431us) server_negotiation.cc:299] Negotiation successful
0807 11:30:57.368689 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260807 11:30:57.373658 4348 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:30:57.381690 4348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:30:57.416251 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.404255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.24.103.66:46293)
0807 11:30:57.404911 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:57.404920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:57.404949 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:30:57.405307 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:57.405317 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:57.406033 (+ 716us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:57.406354 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:57.407401 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.409021 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.412402 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.413838 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.413991 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.414231 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:57.415606 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0807 11:30:57.415821 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":343,"threads_started":1}
I20260807 11:30:57.416477 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.405205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:39641 (local address 127.0.0.1:34120)
0807 11:30:57.405854 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:57.405875 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:57.406002 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:57.407421 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:57.407426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:57.407456 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:57.408048 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.408062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.410198 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.410204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.412810 (+ 2606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.412825 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.415424 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.415432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.415602 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.415634 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 11:30:57.415738 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 11:30:57.415955 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":104,"thread_start_us":349,"threads_started":1}
I20260807 11:30:57.416889 4879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.406502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:41223 (local address 127.0.0.1:50374)
0807 11:30:57.407623 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:57.407646 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:57.407866 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:57.408696 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:57.408700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:57.408731 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:57.409180 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.409193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.411375 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.411381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.413454 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.413470 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.415894 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.415901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.416105 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.416138 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 11:30:57.416239 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 11:30:57.416398 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":4,"thread_start_us":833,"threads_started":1}
I20260807 11:30:57.416317 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.405495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.24.103.67:39641)
0807 11:30:57.406651 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:57.406663 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:57.406706 (+ 43us) server_negotiation.cc:407] Connection header received
0807 11:30:57.406814 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:57.406822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:57.407126 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:57.408277 (+ 1151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:57.408583 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.410045 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.413015 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.414518 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.414715 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.414943 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:57.415806 (+ 863us) server_negotiation.cc:299] Negotiation successful
0807 11:30:57.415969 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":799,"threads_started":1}
I20260807 11:30:57.417232 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.404023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:46293 (local address 127.0.0.1:39890)
0807 11:30:57.404757 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:57.404832 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:57.405037 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:57.406621 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:57.406632 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:57.406672 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:57.407113 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.407135 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.409168 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.409174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.412148 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.412170 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.414991 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.415000 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.415272 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.415353 (+ 81us) client_negotiation.cc:769] Sending connection context
0807 11:30:57.416559 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0807 11:30:57.416706 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":3,"thread_start_us":279,"threads_started":1}
I20260807 11:30:57.417872 4878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.406204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.24.103.65:41223)
0807 11:30:57.407097 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:57.407106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:57.407740 (+ 634us) server_negotiation.cc:407] Connection header received
0807 11:30:57.408094 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:57.408103 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:57.408308 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:57.408464 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:57.409397 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.411127 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.413640 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.415313 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.415481 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.415739 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:57.417242 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0807 11:30:57.417511 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":483,"threads_started":1}
I20260807 11:30:57.420958 4787 tablet_service.cc:1467] Processing CreateTablet for tablet 30e0c0e00bc64fefb9670ddd7f7272da (DEFAULT_TABLE table=test-table [id=2568388da6154b9e8105c9246e8751fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:57.421290 4635 tablet_service.cc:1467] Processing CreateTablet for tablet 30e0c0e00bc64fefb9670ddd7f7272da (DEFAULT_TABLE table=test-table [id=2568388da6154b9e8105c9246e8751fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:57.421659 4487 tablet_service.cc:1467] Processing CreateTablet for tablet 30e0c0e00bc64fefb9670ddd7f7272da (DEFAULT_TABLE table=test-table [id=2568388da6154b9e8105c9246e8751fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:30:57.422394 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e0c0e00bc64fefb9670ddd7f7272da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:57.422768 4635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e0c0e00bc64fefb9670ddd7f7272da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:57.423035 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e0c0e00bc64fefb9670ddd7f7272da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:30:57.434991 4881 tablet_bootstrap.cc:492] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: Bootstrap starting.
I20260807 11:30:57.439126 4881 tablet_bootstrap.cc:654] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:57.439808 4883 tablet_bootstrap.cc:492] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: Bootstrap starting.
I20260807 11:30:57.440835 4881 log.cc:824] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:57.443821 4883 tablet_bootstrap.cc:654] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:57.444152 4882 tablet_bootstrap.cc:492] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: Bootstrap starting.
I20260807 11:30:57.444787 4881 tablet_bootstrap.cc:492] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: No bootstrap required, opened a new log
I20260807 11:30:57.445093 4881 ts_tablet_manager.cc:1397] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260807 11:30:57.445179 4883 log.cc:824] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:57.448030 4882 tablet_bootstrap.cc:654] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: Neither blocks nor log segments found. Creating new log.
I20260807 11:30:57.448751 4883 tablet_bootstrap.cc:492] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: No bootstrap required, opened a new log
I20260807 11:30:57.449069 4883 ts_tablet_manager.cc:1397] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 11:30:57.449471 4882 log.cc:824] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: Log is configured to *not* fsync() on all Append() calls
I20260807 11:30:57.453297 4882 tablet_bootstrap.cc:492] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: No bootstrap required, opened a new log
I20260807 11:30:57.453631 4882 ts_tablet_manager.cc:1397] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260807 11:30:57.454357 4881 raft_consensus.cc:348] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497c3d00c785408088d524bf038d52b0" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46293 } } peers { permanent_uuid: "79697639ad884c7f8ef5c57076be62f9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39641 } } peers { permanent_uuid: "3d77b4c591c74946a97c18cac9ce12ec" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 41223 } }
I20260807 11:30:57.454751 4881 raft_consensus.cc:374] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:57.454860 4881 raft_consensus.cc:729] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79697639ad884c7f8ef5c57076be62f9, State: Initialized, Role: FOLLOWER
I20260807 11:30:57.455543 4881 consensus_queue.cc:260] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497c3d00c785408088d524bf038d52b0" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46293 } } peers { permanent_uuid: "79697639ad884c7f8ef5c57076be62f9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39641 } } peers { permanent_uuid: "3d77b4c591c74946a97c18cac9ce12ec" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 41223 } }
I20260807 11:30:57.456748 4859 heartbeater.cc:503] Master 127.24.103.126:45093 was elected leader, sending a full tablet report...
I20260807 11:30:57.457504 4881 ts_tablet_manager.cc:1428] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260807 11:30:57.461134 4883 raft_consensus.cc:348] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497c3d00c785408088d524bf038d52b0" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46293 } } peers { permanent_uuid: "79697639ad884c7f8ef5c57076be62f9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39641 } } peers { permanent_uuid: "3d77b4c591c74946a97c18cac9ce12ec" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 41223 } }
I20260807 11:30:57.461666 4883 raft_consensus.cc:374] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:57.461796 4883 raft_consensus.cc:729] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497c3d00c785408088d524bf038d52b0, State: Initialized, Role: FOLLOWER
I20260807 11:30:57.462603 4883 consensus_queue.cc:260] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497c3d00c785408088d524bf038d52b0" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46293 } } peers { permanent_uuid: "79697639ad884c7f8ef5c57076be62f9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39641 } } peers { permanent_uuid: "3d77b4c591c74946a97c18cac9ce12ec" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 41223 } }
I20260807 11:30:57.464036 4712 heartbeater.cc:503] Master 127.24.103.126:45093 was elected leader, sending a full tablet report...
I20260807 11:30:57.464946 4883 ts_tablet_manager.cc:1428] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260807 11:30:57.465679 4882 raft_consensus.cc:348] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497c3d00c785408088d524bf038d52b0" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46293 } } peers { permanent_uuid: "79697639ad884c7f8ef5c57076be62f9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39641 } } peers { permanent_uuid: "3d77b4c591c74946a97c18cac9ce12ec" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 41223 } }
I20260807 11:30:57.466105 4882 raft_consensus.cc:374] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:30:57.466228 4882 raft_consensus.cc:729] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d77b4c591c74946a97c18cac9ce12ec, State: Initialized, Role: FOLLOWER
I20260807 11:30:57.466902 4882 consensus_queue.cc:260] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497c3d00c785408088d524bf038d52b0" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46293 } } peers { permanent_uuid: "79697639ad884c7f8ef5c57076be62f9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39641 } } peers { permanent_uuid: "3d77b4c591c74946a97c18cac9ce12ec" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 41223 } }
I20260807 11:30:57.468693 4882 ts_tablet_manager.cc:1428] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
W20260807 11:30:57.496572 4713 tablet.cc:2369] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:30:57.569016 4860 tablet.cc:2369] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:30:57.617556 4888 raft_consensus.cc:482] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:30:57.617913 4888 raft_consensus.cc:504] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497c3d00c785408088d524bf038d52b0" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46293 } } peers { permanent_uuid: "79697639ad884c7f8ef5c57076be62f9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39641 } } peers { permanent_uuid: "3d77b4c591c74946a97c18cac9ce12ec" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 41223 } }
I20260807 11:30:57.619999 4888 leader_election.cc:288] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79697639ad884c7f8ef5c57076be62f9 (127.24.103.67:39641), 3d77b4c591c74946a97c18cac9ce12ec (127.24.103.65:41223)
I20260807 11:30:57.626869 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.620497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:60147 (local address 127.24.103.67:39641)
0807 11:30:57.620800 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:57.620807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:57.621176 (+ 369us) server_negotiation.cc:407] Connection header received
0807 11:30:57.621472 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:57.621482 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:57.621588 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:57.621689 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:57.622307 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.623119 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.624951 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.625868 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.626069 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.626148 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:57.626439 (+ 291us) server_negotiation.cc:299] Negotiation successful
0807 11:30:57.626571 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 11:30:57.627041 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.620350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:39641 (local address 127.24.103.66:60147)
0807 11:30:57.621044 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:57.621068 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:57.621270 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:57.621766 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:57.621771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:57.621798 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:57.622162 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.622172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.623265 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.623272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.624792 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.624805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.626092 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.626099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.626259 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.626283 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 11:30:57.626411 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 11:30:57.626608 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":1,"thread_start_us":390,"threads_started":1}
I20260807 11:30:57.627432 4878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.621495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:55289 (local address 127.24.103.65:41223)
0807 11:30:57.621756 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:57.621762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:57.621960 (+ 198us) server_negotiation.cc:407] Connection header received
0807 11:30:57.622234 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:57.622238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:57.622298 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:57.622399 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:30:57.622938 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.623790 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.625592 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.626449 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.626625 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.626699 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:57.627133 (+ 434us) server_negotiation.cc:299] Negotiation successful
0807 11:30:57.627258 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260807 11:30:57.628191 4892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.621224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:41223 (local address 127.24.103.66:55289)
0807 11:30:57.621853 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0807 11:30:57.621870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:30:57.622037 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:30:57.622412 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:30:57.622416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:30:57.622438 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:30:57.622791 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.622799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.623930 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.623936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.625430 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:30:57.625443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.626653 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.626660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:30:57.626838 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.626860 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 11:30:57.627661 (+ 801us) client_negotiation.cc:240] Negotiation successful
0807 11:30:57.627816 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":24,"thread_start_us":294,"threads_started":1}
I20260807 11:30:57.628157 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e0c0e00bc64fefb9670ddd7f7272da" candidate_uuid: "497c3d00c785408088d524bf038d52b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79697639ad884c7f8ef5c57076be62f9" is_pre_election: true
I20260807 11:30:57.628824 4808 raft_consensus.cc:2454] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 497c3d00c785408088d524bf038d52b0 in term 0.
I20260807 11:30:57.629446 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e0c0e00bc64fefb9670ddd7f7272da" candidate_uuid: "497c3d00c785408088d524bf038d52b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d77b4c591c74946a97c18cac9ce12ec" is_pre_election: true
I20260807 11:30:57.629619 4588 leader_election.cc:302] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 497c3d00c785408088d524bf038d52b0, 79697639ad884c7f8ef5c57076be62f9; no voters:
I20260807 11:30:57.630091 4888 raft_consensus.cc:2781] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:30:57.630048 4508 raft_consensus.cc:2454] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 497c3d00c785408088d524bf038d52b0 in term 0.
I20260807 11:30:57.630218 4888 raft_consensus.cc:482] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:30:57.630393 4888 raft_consensus.cc:3037] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:57.633028 4888 raft_consensus.cc:504] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497c3d00c785408088d524bf038d52b0" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46293 } } peers { permanent_uuid: "79697639ad884c7f8ef5c57076be62f9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39641 } } peers { permanent_uuid: "3d77b4c591c74946a97c18cac9ce12ec" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 41223 } }
I20260807 11:30:57.633706 4888 leader_election.cc:288] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [CANDIDATE]: Term 1 election: Requested vote from peers 79697639ad884c7f8ef5c57076be62f9 (127.24.103.67:39641), 3d77b4c591c74946a97c18cac9ce12ec (127.24.103.65:41223)
I20260807 11:30:57.634202 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e0c0e00bc64fefb9670ddd7f7272da" candidate_uuid: "497c3d00c785408088d524bf038d52b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79697639ad884c7f8ef5c57076be62f9"
I20260807 11:30:57.634289 4508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e0c0e00bc64fefb9670ddd7f7272da" candidate_uuid: "497c3d00c785408088d524bf038d52b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d77b4c591c74946a97c18cac9ce12ec"
I20260807 11:30:57.634397 4808 raft_consensus.cc:3037] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:57.634563 4508 raft_consensus.cc:3037] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:30:57.636715 4808 raft_consensus.cc:2454] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 497c3d00c785408088d524bf038d52b0 in term 1.
I20260807 11:30:57.637142 4588 leader_election.cc:302] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 497c3d00c785408088d524bf038d52b0, 79697639ad884c7f8ef5c57076be62f9; no voters:
I20260807 11:30:57.637306 4508 raft_consensus.cc:2454] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 497c3d00c785408088d524bf038d52b0 in term 1.
I20260807 11:30:57.637470 4888 raft_consensus.cc:2781] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:30:57.638031 4888 raft_consensus.cc:686] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 1 LEADER]: Becoming Leader. State: Replica: 497c3d00c785408088d524bf038d52b0, State: Running, Role: LEADER
I20260807 11:30:57.638520 4888 consensus_queue.cc:237] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497c3d00c785408088d524bf038d52b0" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46293 } } peers { permanent_uuid: "79697639ad884c7f8ef5c57076be62f9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39641 } } peers { permanent_uuid: "3d77b4c591c74946a97c18cac9ce12ec" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 41223 } }
I20260807 11:30:57.643317 4347 catalog_manager.cc:5495] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 497c3d00c785408088d524bf038d52b0 (127.24.103.66). New cstate: current_term: 1 leader_uuid: "497c3d00c785408088d524bf038d52b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497c3d00c785408088d524bf038d52b0" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 46293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79697639ad884c7f8ef5c57076be62f9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d77b4c591c74946a97c18cac9ce12ec" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 41223 } health_report { overall_health: UNKNOWN } } }
I20260807 11:30:57.660470 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.654899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.24.103.66:46293)
0807 11:30:57.655186 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:57.655191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:57.655232 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:30:57.655336 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:57.655339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:57.655425 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:57.655552 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:30:57.655905 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.656805 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.658256 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.658712 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.658824 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.659334 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:30:57.659849 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:30:57.660031 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:57.660089 (+ 58us) server_negotiation.cc:299] Negotiation successful
0807 11:30:57.660177 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260807 11:30:57.667688 4564 tablet.cc:2369] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:30:57.669363 4808 raft_consensus.cc:1261] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9 [term 1 FOLLOWER]: Refusing update from remote peer 497c3d00c785408088d524bf038d52b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:57.669512 4508 raft_consensus.cc:1261] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec [term 1 FOLLOWER]: Refusing update from remote peer 497c3d00c785408088d524bf038d52b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:30:57.670349 4893 consensus_queue.cc:1035] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79697639ad884c7f8ef5c57076be62f9" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:57.670753 4888 consensus_queue.cc:1035] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d77b4c591c74946a97c18cac9ce12ec" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:30:57.683051 4897 mvcc.cc:205] Tried to move back new op lower bound from 7315874847402778624 to 7315874847291486208. Current Snapshot: MvccSnapshot[applied={T|T < 7315874847402778624}]
I20260807 11:30:57.699420 4348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:57.699712 4348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:30:57.703502 4348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4da0d4c46f1a4341b50b91dbb47f167a: Sending DeleteTablet for 3 replicas of tablet 30e0c0e00bc64fefb9670ddd7f7272da
I20260807 11:30:57.704823 4635 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e0c0e00bc64fefb9670ddd7f7272da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:57 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:39890
I20260807 11:30:57.704911 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e0c0e00bc64fefb9670ddd7f7272da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:57 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:34120
I20260807 11:30:57.705001 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e0c0e00bc64fefb9670ddd7f7272da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:30:57 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:50374
I20260807 11:30:57.707449 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 4418
I20260807 11:30:57.717660 4909 tablet_replica.cc:330] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: stopping tablet replica
I20260807 11:30:57.718657 4907 tablet_replica.cc:330] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: stopping tablet replica
I20260807 11:30:57.718972 4909 raft_consensus.cc:2229] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:57.719204 4907 raft_consensus.cc:2229] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:30:57.720009 4907 raft_consensus.cc:2258] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:57.720654 4909 raft_consensus.cc:2258] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:57.720912 4908 tablet_replica.cc:330] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: stopping tablet replica
I20260807 11:30:57.721442 4908 raft_consensus.cc:2229] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:30:57.722038 4908 raft_consensus.cc:2258] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:30:57.723594 4909 ts_tablet_manager.cc:1905] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:57.724376 4908 ts_tablet_manager.cc:1905] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 11:30:57 dist-test-slave-wl2h krb5kdc[4298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:30:57.731117 4909 ts_tablet_manager.cc:1918] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:57.731393 4909 log.cc:1196] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/30e0c0e00bc64fefb9670ddd7f7272da
I20260807 11:30:57.732146 4909 ts_tablet_manager.cc:1939] T 30e0c0e00bc64fefb9670ddd7f7272da P 79697639ad884c7f8ef5c57076be62f9: Deleting consensus metadata
I20260807 11:30:57.732108 4907 ts_tablet_manager.cc:1905] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:30:57.732872 4908 ts_tablet_manager.cc:1918] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:57.733135 4908 log.cc:1196] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/30e0c0e00bc64fefb9670ddd7f7272da
I20260807 11:30:57.733861 4908 ts_tablet_manager.cc:1939] T 30e0c0e00bc64fefb9670ddd7f7272da P 3d77b4c591c74946a97c18cac9ce12ec: Deleting consensus metadata
I20260807 11:30:57.734434 4333 catalog_manager.cc:4841] TS 79697639ad884c7f8ef5c57076be62f9 (127.24.103.67:39641): tablet 30e0c0e00bc64fefb9670ddd7f7272da (table test-table [id=2568388da6154b9e8105c9246e8751fb]) successfully deleted
I20260807 11:30:57.735966 4333 catalog_manager.cc:4841] TS 3d77b4c591c74946a97c18cac9ce12ec (127.24.103.65:41223): tablet 30e0c0e00bc64fefb9670ddd7f7272da (table test-table [id=2568388da6154b9e8105c9246e8751fb]) successfully deleted
I20260807 11:30:57.739132 4878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.710254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50390 (local address 127.24.103.65:41223)
0807 11:30:57.710497 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:57.710506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:57.717897 (+ 7391us) server_negotiation.cc:407] Connection header received
0807 11:30:57.718107 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:57.718295 (+ 188us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:57.718432 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:57.718565 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:57.720906 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.722116 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.724000 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.724642 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.724742 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.730524 (+ 5782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:57.730539 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:57.730558 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:57.730612 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:57.735246 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:57.736203 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:57.736215 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:57.736219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:57.736409 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:57.736915 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:57.736921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:57.736925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:57.737405 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:57.737737 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:57.738246 (+ 509us) server_negotiation.cc:299] Negotiation successful
0807 11:30:57.738504 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:30:57.739923 4557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:57.740556 4907 ts_tablet_manager.cc:1918] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:30:57.740824 4907 log.cc:1196] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/30e0c0e00bc64fefb9670ddd7f7272da
I20260807 11:30:57.741641 4907 ts_tablet_manager.cc:1939] T 30e0c0e00bc64fefb9670ddd7f7272da P 497c3d00c785408088d524bf038d52b0: Deleting consensus metadata
I20260807 11:30:57.745477 4333 catalog_manager.cc:4841] TS 497c3d00c785408088d524bf038d52b0 (127.24.103.66:46293): tablet 30e0c0e00bc64fefb9670ddd7f7272da (table test-table [id=2568388da6154b9e8105c9246e8751fb]) successfully deleted
I20260807 11:30:57.845183 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 4418
I20260807 11:30:57.874171 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 4569
Aug 07 11:30:57 dist-test-slave-wl2h krb5kdc[4298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:30:57.892680 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:57.875051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.24.103.66:46293)
0807 11:30:57.875248 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:57.875253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:57.875273 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:30:57.875625 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:57.875631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:57.875726 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:57.875819 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:57.877579 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.878458 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.880419 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:57.880979 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:57.881078 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:57.886187 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:57.886205 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:57.886221 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:57.886264 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:57.889614 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:57.890416 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:57.890424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:57.890426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:57.890569 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:57.890983 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:57.890987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:57.890989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:57.891437 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:57.891791 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:57.892085 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 11:30:57.892317 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:30:57.893430 4706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:58.009781 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 4569
I20260807 11:30:58.031991 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 4718
Aug 07 11:30:58 dist-test-slave-wl2h krb5kdc[4298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:30:58.049518 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:30:58.032885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34136 (local address 127.24.103.67:39641)
0807 11:30:58.033150 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 11:30:58.033157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:30:58.033416 (+ 259us) server_negotiation.cc:407] Connection header received
0807 11:30:58.033534 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:30:58.033538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:30:58.033614 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:30:58.033722 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:30:58.035858 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:58.036608 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:58.038011 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:30:58.038432 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:30:58.038531 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:30:58.043429 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:30:58.043443 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:30:58.043459 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:30:58.043504 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:30:58.046394 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:58.047223 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:58.047232 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:58.047274 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:58.047456 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:30:58.047937 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:30:58.047941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:30:58.047943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:30:58.048351 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:30:58.048609 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 11:30:58.048920 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 11:30:58.049177 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:30:58.050311 4854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:58.141747 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 4718
I20260807 11:30:58.162227 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 4311
I20260807 11:30:58.163619 4385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:30:58.279596 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 4311
2026-08-07T11:30:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4846 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:30:58 dist-test-slave-wl2h krb5kdc[4919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:30:58 dist-test-slave-wl2h krb5kdc[4919](info): set up 2 sockets
Aug 07 11:30:58 dist-test-slave-wl2h krb5kdc[4919](info): commencing operation
krb5kdc: starting...
W20260807 11:30:59.503574 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:30:59 dist-test-slave-wl2h krb5kdc[4919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:30:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:30:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/rangeradmin_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/rangeradmin_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/rangerlookup_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/rangerlookup_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
I20260807 11:30:59.971181 24989 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taska8aRx7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/postgres -l logfile start
2026-08-07 11:31:02.942 UTC [4952] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 11:31:02.943 UTC [4952] LOG: listening on IPv4 address "127.24.103.84", port 40273
2026-08-07 11:31:02.947 UTC [4952] LOG: listening on Unix socket "/tmp/.s.PGSQL.40273"
2026-08-07 11:31:02.970 UTC [4954] LOG: database system was shut down at 2026-08-07 11:31:01 UTC
2026-08-07 11:31:02.974 UTC [4952] LOG: database system is ready to accept connections
I20260807 11:31:04.048242 24989 mini_postgres.cc:96] Postgres bound to 40273
2026-08-07 11:31:04.074 UTC [4964] FATAL: database "slave" does not exist
127.24.103.84:40273 - accepting connections
I20260807 11:31:04.075133 24989 mini_ranger.cc:161] Starting Ranger...
I20260807 11:31:04.110615 24989 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 11:31:04.714922 24989 mini_ranger.cc:87] Created ranger Postgres database
I20260807 11:31:04.715173 24989 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-admin
2026-08-07 11:31:05,075 [I] DB FLAVOR :POSTGRES
2026-08-07 11:31:05,076 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 11:31:05,076 [I] Checking connection..
2026-08-07 11:31:05,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 11:31:05,338 [I] Checking connection passed.
2026-08-07 11:31:05,338 [I] --------- Verifying version history table ---------
2026-08-07 11:31:05,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:31:05,606 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 11:31:05,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:31:05,879 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 11:31:05,879 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 11:31:05,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 11:31:06.141 UTC [5037] WARNING: there is no transaction in progress
2026-08-07 11:31:06,149 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 11:31:06,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:31:06,417 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 11:31:06,417 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 11:31:06,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 11:31:06,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 11:31:06,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wl2h.c.gcp-upstream.internal','N') ;"
2026-08-07 11:31:07,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 11:31:07,472 [I] Table x_portal_user does not exist in database ranger
2026-08-07 11:31:07,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 11:31:07,751 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 11:31:07,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 11:31:08,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 11:31:08,261 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 11:31:08,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 11:31:09.317 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.337 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.356 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.374 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.396 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.416 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.479 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.487 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.504 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.516 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.529 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.545 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.555 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.563 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.571 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:09.579 UTC [5154] WARNING: there is no transaction in progress
2026-08-07 11:31:10,385 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 11:31:10,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 11:31:10,642 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 11:31:10,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 11:31:10,902 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 11:31:10,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 11:31:11,166 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 11:31:11,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 11:31:11,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 11:31:11,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wl2h.c.gcp-upstream.internal';"
2026-08-07 11:31:11,943 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 11:31:11,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 11:31:12,201 [I] DB_PATCHES have already been applied
I20260807 11:31:12.207780 24989 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-admin:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taska8aRx7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 11:31:12.207901 24989 mini_ranger.cc:193] Using host: 127.24.103.84
I20260807 11:31:12.231228 24989 mini_ranger.cc:227] Ranger admin URL: http://127.24.103.84:55367
Aug 07, 2026 11:31:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 11:31:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 11:31:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 11:31:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 11:31:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 11:31:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 11:31:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 11:31:13.493364 24989 mini_ranger.cc:160] Time spent starting Ranger: real 9.418s user 0.008s sys 0.061s
Aug 07, 2026 11:31:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 11:31:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 11:31:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 11:31:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 11:31:36 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 11:31:36.626159 24989 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/rangerkms_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/rangerkms_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.103.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.103.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 11:31:36 dist-test-slave-wl2h krb5kdc[4919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102296, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 11:31:36.877382 24989 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 11:31:36.907583 24989 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 11:31:37.991971 24989 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 11:31:37.992151 24989 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-kms
I20260807 11:31:37.992323 24989 mini_ranger_kms.cc:227] Using postgres at 127.24.103.84:40273
2026-08-07 11:31:38,530 [I] DB FLAVOR :POSTGRES
2026-08-07 11:31:38,530 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 11:31:38,530 [I] Checking connection
2026-08-07 11:31:38,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 11:31:38,783 [I] connection success
2026-08-07 11:31:38,784 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 11:31:38,784 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 11:31:38,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 11:31:39,081 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 11:31:39,081 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 11:31:39,082 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 11:31:39,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:40273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 11:31:39,366 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 11:31:39.872002 24989 mini_ranger_kms.cc:310] Created kms service
I20260807 11:31:40.045487 24989 mini_ranger_kms.cc:326] Created kudu user
I20260807 11:31:40.098582 24989 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 11:31:40 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102296, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:31:40.505287 24989 mini_ranger_kms.cc:384] Added ranger policy
I20260807 11:31:40.505755 24989 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-kms:/tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 11:31:40.505841 24989 mini_ranger_kms.cc:240] Using host: 127.24.103.84
I20260807 11:31:40.538779 24989 mini_ranger_kms.cc:276] Ranger KMS PID: 5368
I20260807 11:31:40.539024 24989 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.103.84:35155
Aug 07, 2026 11:31:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 11:31:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 11:31:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 11:31:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 11:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 11:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 11:31:42 dist-test-slave-wl2h krb5kdc[4919](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786102302, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:31:43.125200 24989 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.248s user 0.144s sys 0.434s
I20260807 11:31:43.125576 24989 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 11:31:43.125787 24989 mini_ranger_kms.cc:401] 127.24.103.84:35155/kms/v1/keys
Aug 07 11:31:43 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102302, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
Aug 07 11:31:43 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102302, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
Aug 07 11:31:46 dist-test-slave-wl2h krb5kdc[4919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:31:46.184353 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:40175
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:40175
--ranger_config_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:35155/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:31:46.541891 5423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:31:46.542186 5423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:31:46.542279 5423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:31:46.549875 5423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:31:46.549988 5423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:31:46.550033 5423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:31:46.550091 5423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:31:46.550141 5423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:31:46.560544 5423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.103.84:35155/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:40175
--ranger_config_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:40175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:31:46.561863 5423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:31:46.563479 5423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:31:46.575596 5429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:31:46.575654 5431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:31:46.575686 5428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:31:46.576771 5423 server_base.cc:1034] running on GCE node
I20260807 11:31:46.577538 5423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:31:46.579090 5423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:31:46.580323 5423 hybrid_clock.cc:648] HybridClock initialized: now 1786102306580281 us; error 49 us; skew 500 ppm
I20260807 11:31:46.580685 5423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:31:46 dist-test-slave-wl2h krb5kdc[4919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102306, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:31:46.584123 5423 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:31:46.585665 5423 webserver.cc:460] Webserver started at http://127.24.103.126:42723/ using document root <none> and password file <none>
I20260807 11:31:46.586330 5423 fs_manager.cc:359] Metadata directory not provided
I20260807 11:31:46.586465 5423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:31:46.586787 5423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:31:46 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102306, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:31:46.638550 5423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "bae0ba0231284fefbc39ae581d110c18"
format_stamp: "Formatted at 2026-08-07 11:31:46 on dist-test-slave-wl2h"
server_key: "93e7825806618ee53d997fb3ab62c167"
server_key_iv: "507331d582f4d73a7568a17f7ff15714"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:46.639493 5423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "bae0ba0231284fefbc39ae581d110c18"
format_stamp: "Formatted at 2026-08-07 11:31:46 on dist-test-slave-wl2h"
server_key: "93e7825806618ee53d997fb3ab62c167"
server_key_iv: "507331d582f4d73a7568a17f7ff15714"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:46.645115 5423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260807 11:31:46.662011 5438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:31:46.663769 5423 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 11:31:46.663991 5423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "bae0ba0231284fefbc39ae581d110c18"
format_stamp: "Formatted at 2026-08-07 11:31:46 on dist-test-slave-wl2h"
server_key: "93e7825806618ee53d997fb3ab62c167"
server_key_iv: "507331d582f4d73a7568a17f7ff15714"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:46.664264 5423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:31:46.679711 5423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:31:46.683621 5423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:31:46.683943 5423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:31:46.703958 5423 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:40175
I20260807 11:31:46.704000 5498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:40175 every 8 connection(s)
I20260807 11:31:46.705420 5423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:31:46.709455 5500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:31:46.710736 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 5423
I20260807 11:31:46.710970 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:31:46.711928 24989 external_mini_cluster.cc:1428] Setting key b9cda8722c4ba4cf17b355998148eb4d
I20260807 11:31:46.725508 5500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18: Bootstrap starting.
Aug 07 11:31:46 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:31:46.729424 5500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18: Neither blocks nor log segments found. Creating new log.
I20260807 11:31:46.730810 5500 log.cc:824] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18: Log is configured to *not* fsync() on all Append() calls
I20260807 11:31:46.734051 5500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18: No bootstrap required, opened a new log
I20260807 11:31:46.735893 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:46.714317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.24.103.126:40175)
0807 11:31:46.715310 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:46.715325 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:46.715368 (+ 43us) server_negotiation.cc:407] Connection header received
0807 11:31:46.717053 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:46.717081 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:46.717527 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:46.717882 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:31:46.719528 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:46.720895 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:46.722783 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:46.723588 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:46.723745 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:46.728473 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:31:46.728481 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:31:46.728495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:31:46.728534 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:31:46.731808 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:46.732543 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:46.732556 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:46.732562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:46.732724 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:46.733119 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:46.733128 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:46.733130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:46.733686 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:31:46.733994 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:46.734429 (+ 435us) server_negotiation.cc:299] Negotiation successful
0807 11:31:46.734852 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":436,"threads_started":1}
I20260807 11:31:46.742993 5500 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0ba0231284fefbc39ae581d110c18" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40175 } }
I20260807 11:31:46.743571 5500 raft_consensus.cc:374] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:31:46.743687 5500 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae0ba0231284fefbc39ae581d110c18, State: Initialized, Role: FOLLOWER
I20260807 11:31:46.744274 5500 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0ba0231284fefbc39ae581d110c18" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40175 } }
I20260807 11:31:46.744509 5500 raft_consensus.cc:388] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:31:46.744611 5500 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:31:46.744740 5500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:31:46.747056 5500 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0ba0231284fefbc39ae581d110c18" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40175 } }
I20260807 11:31:46.747527 5500 leader_election.cc:302] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bae0ba0231284fefbc39ae581d110c18; no voters:
I20260807 11:31:46.748114 5500 leader_election.cc:288] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:31:46.748227 5506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:31:46.749589 5506 raft_consensus.cc:686] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [term 1 LEADER]: Becoming Leader. State: Replica: bae0ba0231284fefbc39ae581d110c18, State: Running, Role: LEADER
I20260807 11:31:46.750126 5506 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0ba0231284fefbc39ae581d110c18" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40175 } }
I20260807 11:31:46.750280 5500 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:31:46.753084 5507 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bae0ba0231284fefbc39ae581d110c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0ba0231284fefbc39ae581d110c18" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40175 } } }
I20260807 11:31:46.753058 5508 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae0ba0231284fefbc39ae581d110c18. Latest consensus state: current_term: 1 leader_uuid: "bae0ba0231284fefbc39ae581d110c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0ba0231284fefbc39ae581d110c18" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40175 } } }
I20260807 11:31:46.753551 5507 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [sys.catalog]: This master's current role is: LEADER
I20260807 11:31:46.753551 5508 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18 [sys.catalog]: This master's current role is: LEADER
I20260807 11:31:46.754536 5513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:31:46.764587 5513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:31:46.769022 5500 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 11:31:46.785246 5513 catalog_manager.cc:1347] Generated new cluster ID: 7a8f26c7e34044d2b04ed32e0290e713
I20260807 11:31:46.785547 5513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:31:46.814810 5513 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:31:46.816605 5513 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:31:46.835929 5513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18: Generated new TSK 0
I20260807 11:31:46.837138 5513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:31:47.584870 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:35155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40175
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:31:47 dist-test-slave-wl2h krb5kdc[4919](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786102307, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 11:31:48.056041 5548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:31:48.056488 5548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:31:48.056703 5548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:31:48.068624 5548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:31:48.068861 5548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:31:48.069104 5548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:31:48.085418 5548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.103.84:35155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:31:48.087376 5548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:31:48.089175 5548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:31:48.107167 5560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:31:48.107573 5562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:31:48.108042 5559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:31:48.108947 5548 server_base.cc:1034] running on GCE node
I20260807 11:31:48.109743 5548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:31:48.111052 5548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:31:48.112286 5548 hybrid_clock.cc:648] HybridClock initialized: now 1786102308112216 us; error 55 us; skew 500 ppm
I20260807 11:31:48.112661 5548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:31:48 dist-test-slave-wl2h krb5kdc[4919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102308, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:31:48.121074 5548 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:31:48.122823 5548 webserver.cc:460] Webserver started at http://127.24.103.65:33603/ using document root <none> and password file <none>
I20260807 11:31:48.123672 5548 fs_manager.cc:359] Metadata directory not provided
I20260807 11:31:48.123812 5548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:31:48.124148 5548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:31:48 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102308, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:31:48.152426 5548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "81587f0016414f1291b4e31c813f8a10"
format_stamp: "Formatted at 2026-08-07 11:31:48 on dist-test-slave-wl2h"
server_key: "f3082fabec067f7bb67affa79a008282"
server_key_iv: "69d51112969b535e96f79b6d6d3bb9ef"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:48.153738 5548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "81587f0016414f1291b4e31c813f8a10"
format_stamp: "Formatted at 2026-08-07 11:31:48 on dist-test-slave-wl2h"
server_key: "f3082fabec067f7bb67affa79a008282"
server_key_iv: "69d51112969b535e96f79b6d6d3bb9ef"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:48.161979 5548 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260807 11:31:48.201550 5570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:31:48.204209 5548 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.008s sys 0.012s
I20260807 11:31:48.204425 5548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "81587f0016414f1291b4e31c813f8a10"
format_stamp: "Formatted at 2026-08-07 11:31:48 on dist-test-slave-wl2h"
server_key: "f3082fabec067f7bb67affa79a008282"
server_key_iv: "69d51112969b535e96f79b6d6d3bb9ef"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:48.204815 5548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:31:48.223208 5548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:31:48.229244 5548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:31:48.229600 5548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:31:48.230887 5548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:31:48.235898 5548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:31:48.236044 5548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:31:48.236199 5548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:31:48.236282 5548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:31:48 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102307, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:31:48.341897 5702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:45623 every 8 connection(s)
I20260807 11:31:48.341887 5548 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:45623
I20260807 11:31:48.343801 5548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:31:48.344455 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 5548
I20260807 11:31:48.344667 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:31:48.345465 24989 external_mini_cluster.cc:1428] Setting key d9220581c62c55519c50d58db02aa8a8
Aug 07 11:31:48 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102308, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
I20260807 11:31:48.437758 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:48.395318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40175 (local address 127.24.103.65:56129)
0807 11:31:48.412577 (+ 17259us) negotiation.cc:107] Waiting for socket to connect
0807 11:31:48.412642 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 11:31:48.415192 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:31:48.416248 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:31:48.416263 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:31:48.416966 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:31:48.417914 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:48.417941 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:48.419710 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:48.419716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:48.421950 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:48.421967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:48.423432 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:48.423441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:48.423775 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:48.424625 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:31:48.424665 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:31:48.427716 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:31:48.432220 (+ 4504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:31:48.432231 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:31:48.432260 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:31:48.433022 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:31:48.433561 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:31:48.433566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:31:48.433570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:31:48.433864 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:31:48.434744 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:31:48.434754 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:31:48.435309 (+ 555us) client_negotiation.cc:769] Sending connection context
0807 11:31:48.435655 (+ 346us) client_negotiation.cc:240] Negotiation successful
0807 11:31:48.436248 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16906,"thread_start_us":16448,"threads_started":1}
I20260807 11:31:48.439148 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:48.396360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:56129 (local address 127.24.103.126:40175)
0807 11:31:48.411944 (+ 15584us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:48.411951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:48.415604 (+ 3653us) server_negotiation.cc:407] Connection header received
0807 11:31:48.415684 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:48.415688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:48.415792 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:48.415903 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:31:48.418201 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:48.419483 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:48.422180 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:48.422873 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:48.423060 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:48.428032 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:31:48.428040 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:31:48.428044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:31:48.428090 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:31:48.432043 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:48.433199 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:48.433205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:48.433208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:48.433394 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:48.434025 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:48.434030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:48.434035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:48.434495 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:31:48.434658 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:48.438426 (+ 3768us) server_negotiation.cc:299] Negotiation successful
0807 11:31:48.438712 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15413,"thread_start_us":15718,"threads_started":1}
I20260807 11:31:48.441704 5703 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40175
I20260807 11:31:48.442057 5703 heartbeater.cc:464] Registering TS with master...
I20260807 11:31:48.442994 5703 heartbeater.cc:511] Master 127.24.103.126:40175 requested a full tablet report, sending...
I20260807 11:31:48.445791 5457 ts_manager.cc:194] Registered new tserver with Master: 81587f0016414f1291b4e31c813f8a10 (127.24.103.65:45623)
I20260807 11:31:48.448042 5457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:56129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
W20260807 11:31:48.474282 24989 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.24.103.66: real 0.126s user 0.001s sys 0.024s
I20260807 11:31:48.475255 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:35155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40175
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:31:48 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102307, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
W20260807 11:31:48.955322 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:31:48.955858 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:31:48.956072 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:31:48.968000 5711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:31:48.968269 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:31:48.968529 5711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:31:48.984886 5711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.103.84:35155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:31:48.986980 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:31:48.988891 5711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:31:49.024129 5719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:31:49.028390 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:31:49.031134 5722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:31:49.031702 5711 server_base.cc:1034] running on GCE node
I20260807 11:31:49.032765 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:31:49.039140 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:31:49.040547 5711 hybrid_clock.cc:648] HybridClock initialized: now 1786102309040435 us; error 112 us; skew 500 ppm
I20260807 11:31:49.041143 5711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:31:49 dist-test-slave-wl2h krb5kdc[4919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:31:49.045709 5711 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:31:49.066879 5711 webserver.cc:460] Webserver started at http://127.24.103.66:42913/ using document root <none> and password file <none>
I20260807 11:31:49.068077 5711 fs_manager.cc:359] Metadata directory not provided
I20260807 11:31:49.068320 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:31:49.068794 5711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:31:49 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:31:49.095193 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "36ac14d7cac94c2697afbefda383cd68"
format_stamp: "Formatted at 2026-08-07 11:31:49 on dist-test-slave-wl2h"
server_key: "b19b0a92ad87be21add0e371c4ac1ec6"
server_key_iv: "ccbde5da0657c8ef50b9303991298b76"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:49.096489 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "36ac14d7cac94c2697afbefda383cd68"
format_stamp: "Formatted at 2026-08-07 11:31:49 on dist-test-slave-wl2h"
server_key: "b19b0a92ad87be21add0e371c4ac1ec6"
server_key_iv: "ccbde5da0657c8ef50b9303991298b76"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:49.115666 5711 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.009s sys 0.008s
I20260807 11:31:49.148046 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:31:49.150290 5711 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.005s sys 0.006s
I20260807 11:31:49.150604 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "36ac14d7cac94c2697afbefda383cd68"
format_stamp: "Formatted at 2026-08-07 11:31:49 on dist-test-slave-wl2h"
server_key: "b19b0a92ad87be21add0e371c4ac1ec6"
server_key_iv: "ccbde5da0657c8ef50b9303991298b76"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:49.151118 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:31:49.207067 5711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:31:49.211629 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:31:49.211956 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:31:49.213075 5711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:31:49.214699 5711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:31:49.214802 5711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:31:49.214959 5711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:31:49.215065 5711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:31:49.269872 5711 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:36271
I20260807 11:31:49.269892 5863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:36271 every 8 connection(s)
I20260807 11:31:49.271304 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:31:49.276834 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 5711
I20260807 11:31:49.277029 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:31:49.277518 24989 external_mini_cluster.cc:1428] Setting key 9bb120b887ad940b87fac95bee8634ec
Aug 07 11:31:49 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:31:49.297914 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:49.274269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:45247 (local address 127.24.103.126:40175)
0807 11:31:49.274821 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:49.274828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:49.275888 (+ 1060us) server_negotiation.cc:407] Connection header received
0807 11:31:49.278508 (+ 2620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:49.278513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:49.278650 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:49.278788 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:31:49.280806 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.281878 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.284782 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.285442 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.285595 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:49.290319 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:31:49.290329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:31:49.290337 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:31:49.290388 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:31:49.293644 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:49.294661 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:49.294670 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:49.294677 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:49.294848 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:49.295426 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:49.295435 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:49.295440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:49.295933 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:31:49.296091 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:49.297062 (+ 971us) server_negotiation.cc:299] Negotiation successful
0807 11:31:49.297369 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":268,"threads_started":1}
I20260807 11:31:49.299155 5869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:49.274817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40175 (local address 127.24.103.66:45247)
0807 11:31:49.275701 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0807 11:31:49.275752 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 11:31:49.278205 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:31:49.279099 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:31:49.279110 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:31:49.279732 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:31:49.280582 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:49.280609 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.282066 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.282075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:49.284569 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:49.284605 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.285705 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.285714 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:49.286050 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:49.286931 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:31:49.286969 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:31:49.290038 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:31:49.293801 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:31:49.293811 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:31:49.293829 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:31:49.294460 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:31:49.294993 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:31:49.294999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:31:49.295002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:31:49.295283 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:31:49.296101 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:31:49.296110 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:31:49.296574 (+ 464us) client_negotiation.cc:769] Sending connection context
0807 11:31:49.296915 (+ 341us) client_negotiation.cc:240] Negotiation successful
0807 11:31:49.297506 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":367,"threads_started":1}
I20260807 11:31:49.300915 5864 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40175
I20260807 11:31:49.301265 5864 heartbeater.cc:464] Registering TS with master...
I20260807 11:31:49.302166 5864 heartbeater.cc:511] Master 127.24.103.126:40175 requested a full tablet report, sending...
I20260807 11:31:49.304025 5457 ts_manager.cc:194] Registered new tserver with Master: 36ac14d7cac94c2697afbefda383cd68 (127.24.103.66:36271)
I20260807 11:31:49.305226 5457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:45247
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:31:49.351303 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:35155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40175
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:31:49.452117 5703 heartbeater.cc:503] Master 127.24.103.126:40175 was elected leader, sending a full tablet report...
W20260807 11:31:49.713944 5872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:31:49.714249 5872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:31:49.714433 5872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:31:49.722436 5872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:31:49.722561 5872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:31:49.722811 5872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:31:49.733418 5872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.103.84:35155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:31:49.734792 5872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:31:49.736303 5872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:31:49.750123 5879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:31:49.750224 5881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:31:49.750224 5878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:31:49.756655 5872 server_base.cc:1034] running on GCE node
I20260807 11:31:49.757488 5872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:31:49.758539 5872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:31:49.759750 5872 hybrid_clock.cc:648] HybridClock initialized: now 1786102309759710 us; error 36 us; skew 500 ppm
I20260807 11:31:49.760092 5872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:31:49 dist-test-slave-wl2h krb5kdc[4919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:31:49.763799 5872 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:31:49.765430 5872 webserver.cc:460] Webserver started at http://127.24.103.67:41829/ using document root <none> and password file <none>
I20260807 11:31:49.766108 5872 fs_manager.cc:359] Metadata directory not provided
I20260807 11:31:49.766213 5872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:31:49.766479 5872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:31:49 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:31:49.780990 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "378055cb28e2485a84fc4225671fa425"
format_stamp: "Formatted at 2026-08-07 11:31:49 on dist-test-slave-wl2h"
server_key: "99396cc61bcc956692ee140508625639"
server_key_iv: "19d6a4f2836325f7f233a7f1e92c9632"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:49.781948 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "378055cb28e2485a84fc4225671fa425"
format_stamp: "Formatted at 2026-08-07 11:31:49 on dist-test-slave-wl2h"
server_key: "99396cc61bcc956692ee140508625639"
server_key_iv: "19d6a4f2836325f7f233a7f1e92c9632"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:49.788134 5872 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260807 11:31:49.806793 5889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:31:49.808771 5872 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 11:31:49.808987 5872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "378055cb28e2485a84fc4225671fa425"
format_stamp: "Formatted at 2026-08-07 11:31:49 on dist-test-slave-wl2h"
server_key: "99396cc61bcc956692ee140508625639"
server_key_iv: "19d6a4f2836325f7f233a7f1e92c9632"
server_key_version: "kuduclusterkey@0"
I20260807 11:31:49.809295 5872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:31:49.837706 5872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:31:49.841553 5872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:31:49.841831 5872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:31:49.842792 5872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:31:49.844224 5872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:31:49.844305 5872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:31:49.844391 5872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:31:49.844436 5872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:31:49.880261 5872 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:44765
I20260807 11:31:49.880316 6017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:44765 every 8 connection(s)
I20260807 11:31:49.881613 5872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:31:49.883656 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 5872
I20260807 11:31:49.883857 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:31:49.884450 24989 external_mini_cluster.cc:1428] Setting key b31346ec31e6bf4cb8c43e2f22487c13
Aug 07 11:31:49 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:31:49.902897 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:49.884403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:53427 (local address 127.24.103.126:40175)
0807 11:31:49.884951 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:49.884955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:49.886125 (+ 1170us) server_negotiation.cc:407] Connection header received
0807 11:31:49.888536 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:49.888540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:49.888611 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:49.888705 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:31:49.890199 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.890931 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.892622 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.893099 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.893207 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:49.896652 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:31:49.896657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:31:49.896661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:31:49.896694 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:31:49.899789 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:49.900779 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:49.900788 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:49.900792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:49.900945 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:49.901399 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:49.901402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:49.901404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:49.901677 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:31:49.901791 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:49.902451 (+ 660us) server_negotiation.cc:299] Negotiation successful
0807 11:31:49.902614 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":289,"threads_started":1}
I20260807 11:31:49.904007 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:49.885008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40175 (local address 127.24.103.67:53427)
0807 11:31:49.885934 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0807 11:31:49.885999 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 11:31:49.888250 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:31:49.888908 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:31:49.888916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:31:49.889377 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:31:49.890006 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:49.890025 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.891153 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.891156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:49.892445 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:49.892457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.893323 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.893329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:49.893515 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:49.894214 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:31:49.894300 (+ 86us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:31:49.896433 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:31:49.899940 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:31:49.899947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:31:49.899960 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:31:49.900592 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:31:49.901049 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:31:49.901053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:31:49.901055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:31:49.901288 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:31:49.901787 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:31:49.901793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:31:49.902101 (+ 308us) client_negotiation.cc:769] Sending connection context
0807 11:31:49.902358 (+ 257us) client_negotiation.cc:240] Negotiation successful
0807 11:31:49.902730 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":359,"threads_started":1}
I20260807 11:31:49.905349 6018 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40175
I20260807 11:31:49.905710 6018 heartbeater.cc:464] Registering TS with master...
I20260807 11:31:49.906474 6018 heartbeater.cc:511] Master 127.24.103.126:40175 requested a full tablet report, sending...
I20260807 11:31:49.907961 5457 ts_manager.cc:194] Registered new tserver with Master: 378055cb28e2485a84fc4225671fa425 (127.24.103.67:44765)
I20260807 11:31:49.908771 5457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:53427
I20260807 11:31:49.912947 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:31:49.931699 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:49.918144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.24.103.126:40175)
0807 11:31:49.918483 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:49.918491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:49.918868 (+ 377us) server_negotiation.cc:407] Connection header received
0807 11:31:49.919213 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:49.919217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:49.919309 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:49.919402 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:31:49.920920 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.921674 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.923230 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.923676 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.923781 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:49.926437 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:31:49.926442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:31:49.926445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:31:49.926481 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:31:49.929305 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:49.929995 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:49.929998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:49.930001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:49.930121 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:49.930497 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:49.930500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:49.930502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:49.930766 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:31:49.930945 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:49.931292 (+ 347us) server_negotiation.cc:299] Negotiation successful
0807 11:31:49.931465 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260807 11:31:49.936031 5457 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:31:49.943619 5457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:31:49.981920 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:49.957593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.24.103.66:36271)
0807 11:31:49.971263 (+ 13670us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:49.971276 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:49.971316 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:31:49.971442 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:49.971453 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:49.971841 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:49.972177 (+ 336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:31:49.973187 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.975909 (+ 2722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.978664 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.980326 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.980470 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:49.980684 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:49.981393 (+ 709us) server_negotiation.cc:299] Negotiation successful
0807 11:31:49.981579 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13443,"thread_start_us":13263,"threads_started":1}
I20260807 11:31:49.982004 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:49.956873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:36271 (local address 127.0.0.1:47740)
0807 11:31:49.969308 (+ 12435us) negotiation.cc:107] Waiting for socket to connect
0807 11:31:49.969370 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 11:31:49.969603 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:31:49.972197 (+ 2594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:31:49.972207 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:31:49.972281 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:31:49.972898 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:49.972917 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.976069 (+ 3152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.976077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:49.978444 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:49.978460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.980844 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.980850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:49.981115 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:49.981176 (+ 61us) client_negotiation.cc:769] Sending connection context
0807 11:31:49.981329 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 11:31:49.981547 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12047,"mutex_wait_us":37,"thread_start_us":12992,"threads_started":1}
I20260807 11:31:49.983304 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:49.957659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:44765 (local address 127.0.0.1:33664)
0807 11:31:49.973780 (+ 16121us) negotiation.cc:107] Waiting for socket to connect
0807 11:31:49.973806 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:31:49.973892 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:31:49.975244 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:31:49.975249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:31:49.975286 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:31:49.975698 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:49.975713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.977640 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.977645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:49.979951 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:49.979968 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.982439 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.982451 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:49.982646 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:49.982681 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:31:49.982774 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 11:31:49.982907 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15993,"thread_start_us":15852,"threads_started":1}
I20260807 11:31:49.983901 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:49.958258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.24.103.65:45623)
0807 11:31:49.976096 (+ 17838us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:49.976107 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:49.976137 (+ 30us) server_negotiation.cc:407] Connection header received
0807 11:31:49.976237 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:49.976244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:49.976511 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:49.976717 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:31:49.977435 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.978808 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.981214 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.982665 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.982841 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:49.983087 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:49.983359 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 11:31:49.983529 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17630,"thread_start_us":17426,"threads_started":1}
I20260807 11:31:49.984059 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:49.958888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.24.103.67:44765)
0807 11:31:49.974553 (+ 15665us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:49.974562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:49.974599 (+ 37us) server_negotiation.cc:407] Connection header received
0807 11:31:49.974699 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:49.974707 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:49.974998 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:49.975260 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:31:49.975927 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.977492 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.980169 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.981642 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.981777 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:49.981993 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:49.983579 (+ 1586us) server_negotiation.cc:299] Negotiation successful
0807 11:31:49.983755 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15470,"thread_start_us":15273,"threads_started":1}
I20260807 11:31:49.984678 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:49.957256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45623 (local address 127.0.0.1:36752)
0807 11:31:49.973062 (+ 15806us) negotiation.cc:107] Waiting for socket to connect
0807 11:31:49.973088 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:31:49.973238 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:31:49.976735 (+ 3497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:31:49.976741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:31:49.976768 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:31:49.977204 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:49.977217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.978957 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.978964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:49.981056 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:49.981069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:49.982912 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:49.982918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:49.983135 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:49.983170 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:31:49.983302 (+ 132us) client_negotiation.cc:240] Negotiation successful
0807 11:31:49.983440 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15613,"mutex_wait_us":6,"thread_start_us":15409,"threads_started":1}
I20260807 11:31:49.986604 5791 tablet_service.cc:1467] Processing CreateTablet for tablet 90740e5fbd3a4e5aaf9a17aee811dd78 (DEFAULT_TABLE table=test-table [id=ae802239d5b34e8fa275fc80f0496ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:31:49.987604 5943 tablet_service.cc:1467] Processing CreateTablet for tablet 90740e5fbd3a4e5aaf9a17aee811dd78 (DEFAULT_TABLE table=test-table [id=ae802239d5b34e8fa275fc80f0496ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:31:49.988243 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90740e5fbd3a4e5aaf9a17aee811dd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:31:49.988296 5628 tablet_service.cc:1467] Processing CreateTablet for tablet 90740e5fbd3a4e5aaf9a17aee811dd78 (DEFAULT_TABLE table=test-table [id=ae802239d5b34e8fa275fc80f0496ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:31:49.989032 5943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90740e5fbd3a4e5aaf9a17aee811dd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:31:49.989713 5628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90740e5fbd3a4e5aaf9a17aee811dd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:31:50.000624 6042 tablet_bootstrap.cc:492] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: Bootstrap starting.
I20260807 11:31:50.004009 6043 tablet_bootstrap.cc:492] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68: Bootstrap starting.
I20260807 11:31:50.004483 6042 tablet_bootstrap.cc:654] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: Neither blocks nor log segments found. Creating new log.
I20260807 11:31:50.005925 6044 tablet_bootstrap.cc:492] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: Bootstrap starting.
I20260807 11:31:50.005980 6042 log.cc:824] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: Log is configured to *not* fsync() on all Append() calls
I20260807 11:31:50.008503 6043 tablet_bootstrap.cc:654] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68: Neither blocks nor log segments found. Creating new log.
I20260807 11:31:50.009783 6044 tablet_bootstrap.cc:654] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: Neither blocks nor log segments found. Creating new log.
I20260807 11:31:50.010025 6043 log.cc:824] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68: Log is configured to *not* fsync() on all Append() calls
I20260807 11:31:50.010856 6042 tablet_bootstrap.cc:492] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: No bootstrap required, opened a new log
I20260807 11:31:50.010963 6044 log.cc:824] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: Log is configured to *not* fsync() on all Append() calls
I20260807 11:31:50.011252 6042 ts_tablet_manager.cc:1397] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260807 11:31:50.014060 6044 tablet_bootstrap.cc:492] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: No bootstrap required, opened a new log
I20260807 11:31:50.014426 6044 ts_tablet_manager.cc:1397] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:31:50.023092 6044 raft_consensus.cc:348] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ac14d7cac94c2697afbefda383cd68" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36271 } } peers { permanent_uuid: "81587f0016414f1291b4e31c813f8a10" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45623 } } peers { permanent_uuid: "378055cb28e2485a84fc4225671fa425" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44765 } }
I20260807 11:31:50.023808 6044 raft_consensus.cc:374] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:31:50.023968 6044 raft_consensus.cc:729] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81587f0016414f1291b4e31c813f8a10, State: Initialized, Role: FOLLOWER
I20260807 11:31:50.024863 6044 consensus_queue.cc:260] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ac14d7cac94c2697afbefda383cd68" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36271 } } peers { permanent_uuid: "81587f0016414f1291b4e31c813f8a10" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45623 } } peers { permanent_uuid: "378055cb28e2485a84fc4225671fa425" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44765 } }
I20260807 11:31:50.025084 6042 raft_consensus.cc:348] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ac14d7cac94c2697afbefda383cd68" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36271 } } peers { permanent_uuid: "81587f0016414f1291b4e31c813f8a10" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45623 } } peers { permanent_uuid: "378055cb28e2485a84fc4225671fa425" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44765 } }
I20260807 11:31:50.025588 6042 raft_consensus.cc:374] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:31:50.025743 6042 raft_consensus.cc:729] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378055cb28e2485a84fc4225671fa425, State: Initialized, Role: FOLLOWER
I20260807 11:31:50.026635 6043 tablet_bootstrap.cc:492] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68: No bootstrap required, opened a new log
I20260807 11:31:50.026566 6042 consensus_queue.cc:260] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ac14d7cac94c2697afbefda383cd68" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36271 } } peers { permanent_uuid: "81587f0016414f1291b4e31c813f8a10" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45623 } } peers { permanent_uuid: "378055cb28e2485a84fc4225671fa425" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44765 } }
I20260807 11:31:50.027092 6043 ts_tablet_manager.cc:1397] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.019s
I20260807 11:31:50.027886 6044 ts_tablet_manager.cc:1428] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260807 11:31:50.029883 6018 heartbeater.cc:503] Master 127.24.103.126:40175 was elected leader, sending a full tablet report...
I20260807 11:31:50.030557 6042 ts_tablet_manager.cc:1428] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20260807 11:31:50.036338 6043 raft_consensus.cc:348] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ac14d7cac94c2697afbefda383cd68" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36271 } } peers { permanent_uuid: "81587f0016414f1291b4e31c813f8a10" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45623 } } peers { permanent_uuid: "378055cb28e2485a84fc4225671fa425" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44765 } }
I20260807 11:31:50.036939 6043 raft_consensus.cc:374] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:31:50.037074 6043 raft_consensus.cc:729] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ac14d7cac94c2697afbefda383cd68, State: Initialized, Role: FOLLOWER
I20260807 11:31:50.037773 6043 consensus_queue.cc:260] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ac14d7cac94c2697afbefda383cd68" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36271 } } peers { permanent_uuid: "81587f0016414f1291b4e31c813f8a10" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45623 } } peers { permanent_uuid: "378055cb28e2485a84fc4225671fa425" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44765 } }
I20260807 11:31:50.038848 5864 heartbeater.cc:503] Master 127.24.103.126:40175 was elected leader, sending a full tablet report...
I20260807 11:31:50.039549 6043 ts_tablet_manager.cc:1428] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260807 11:31:50.080492 6051 raft_consensus.cc:482] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:31:50.080904 6051 raft_consensus.cc:504] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ac14d7cac94c2697afbefda383cd68" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36271 } } peers { permanent_uuid: "81587f0016414f1291b4e31c813f8a10" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45623 } } peers { permanent_uuid: "378055cb28e2485a84fc4225671fa425" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44765 } }
I20260807 11:31:50.083081 6051 leader_election.cc:288] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81587f0016414f1291b4e31c813f8a10 (127.24.103.65:45623), 378055cb28e2485a84fc4225671fa425 (127.24.103.67:44765)
I20260807 11:31:50.093322 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:50.083596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:46761 (local address 127.24.103.67:44765)
0807 11:31:50.083828 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:50.083835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:50.084760 (+ 925us) server_negotiation.cc:407] Connection header received
0807 11:31:50.085013 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:50.085019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:50.085157 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:50.085251 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:31:50.086181 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.087349 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.090547 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.091894 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.092080 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:50.092155 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:50.092869 (+ 714us) server_negotiation.cc:299] Negotiation successful
0807 11:31:50.093014 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:31:50.093287 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:50.083641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:47779 (local address 127.24.103.65:45623)
0807 11:31:50.083870 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:50.083879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:50.084759 (+ 880us) server_negotiation.cc:407] Connection header received
0807 11:31:50.084971 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:50.084977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:50.085074 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:50.085194 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:31:50.086149 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.087338 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.090551 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.091865 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.092075 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:50.092160 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:50.092869 (+ 709us) server_negotiation.cc:299] Negotiation successful
0807 11:31:50.093014 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:31:50.093545 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:50.083407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:44765 (local address 127.24.103.66:46761)
0807 11:31:50.084590 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0807 11:31:50.084619 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 11:31:50.084859 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:31:50.085269 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:31:50.085273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:31:50.085305 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:31:50.085921 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:50.085941 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.087506 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.087512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:50.090316 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:50.090333 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.092241 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.092249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:50.092506 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:50.092653 (+ 147us) client_negotiation.cc:769] Sending connection context
0807 11:31:50.092767 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 11:31:50.092993 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":275,"spinlock_wait_cycles":22912,"thread_start_us":674,"threads_started":1}
I20260807 11:31:50.093572 6052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:50.083344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45623 (local address 127.24.103.66:47779)
0807 11:31:50.084590 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0807 11:31:50.084613 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:31:50.084783 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:31:50.085265 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:31:50.085271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:31:50.085330 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:31:50.085969 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:50.085985 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.087507 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.087512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:50.090316 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:31:50.090340 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.092241 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.092249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:31:50.092508 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:50.092651 (+ 143us) client_negotiation.cc:769] Sending connection context
0807 11:31:50.092767 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 11:31:50.092993 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":395,"spinlock_wait_cycles":32640,"thread_start_us":717,"threads_started":1}
I20260807 11:31:50.094838 5652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90740e5fbd3a4e5aaf9a17aee811dd78" candidate_uuid: "36ac14d7cac94c2697afbefda383cd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81587f0016414f1291b4e31c813f8a10" is_pre_election: true
I20260807 11:31:50.094870 5967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90740e5fbd3a4e5aaf9a17aee811dd78" candidate_uuid: "36ac14d7cac94c2697afbefda383cd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378055cb28e2485a84fc4225671fa425" is_pre_election: true
I20260807 11:31:50.095454 5652 raft_consensus.cc:2454] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ac14d7cac94c2697afbefda383cd68 in term 0.
I20260807 11:31:50.095474 5967 raft_consensus.cc:2454] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ac14d7cac94c2697afbefda383cd68 in term 0.
I20260807 11:31:50.096210 5737 leader_election.cc:302] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36ac14d7cac94c2697afbefda383cd68, 378055cb28e2485a84fc4225671fa425, 81587f0016414f1291b4e31c813f8a10; no voters:
I20260807 11:31:50.096706 6051 raft_consensus.cc:2781] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:31:50.096946 6051 raft_consensus.cc:482] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:31:50.097054 6051 raft_consensus.cc:3037] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:31:50.099630 6051 raft_consensus.cc:504] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ac14d7cac94c2697afbefda383cd68" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36271 } } peers { permanent_uuid: "81587f0016414f1291b4e31c813f8a10" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45623 } } peers { permanent_uuid: "378055cb28e2485a84fc4225671fa425" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44765 } }
I20260807 11:31:50.100311 6051 leader_election.cc:288] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [CANDIDATE]: Term 1 election: Requested vote from peers 81587f0016414f1291b4e31c813f8a10 (127.24.103.65:45623), 378055cb28e2485a84fc4225671fa425 (127.24.103.67:44765)
I20260807 11:31:50.100854 5652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90740e5fbd3a4e5aaf9a17aee811dd78" candidate_uuid: "36ac14d7cac94c2697afbefda383cd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81587f0016414f1291b4e31c813f8a10"
I20260807 11:31:50.100894 5967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90740e5fbd3a4e5aaf9a17aee811dd78" candidate_uuid: "36ac14d7cac94c2697afbefda383cd68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378055cb28e2485a84fc4225671fa425"
I20260807 11:31:50.101083 5652 raft_consensus.cc:3037] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:31:50.101114 5967 raft_consensus.cc:3037] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:31:50.103600 5967 raft_consensus.cc:2454] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ac14d7cac94c2697afbefda383cd68 in term 1.
I20260807 11:31:50.103792 5652 raft_consensus.cc:2454] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ac14d7cac94c2697afbefda383cd68 in term 1.
I20260807 11:31:50.104101 5739 leader_election.cc:302] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ac14d7cac94c2697afbefda383cd68, 378055cb28e2485a84fc4225671fa425; no voters:
I20260807 11:31:50.104482 6051 raft_consensus.cc:2781] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:31:50.104966 6051 raft_consensus.cc:686] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 1 LEADER]: Becoming Leader. State: Replica: 36ac14d7cac94c2697afbefda383cd68, State: Running, Role: LEADER
I20260807 11:31:50.105458 6051 consensus_queue.cc:237] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ac14d7cac94c2697afbefda383cd68" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36271 } } peers { permanent_uuid: "81587f0016414f1291b4e31c813f8a10" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45623 } } peers { permanent_uuid: "378055cb28e2485a84fc4225671fa425" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44765 } }
I20260807 11:31:50.110657 5455 catalog_manager.cc:5495] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ac14d7cac94c2697afbefda383cd68 (127.24.103.66). New cstate: current_term: 1 leader_uuid: "36ac14d7cac94c2697afbefda383cd68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ac14d7cac94c2697afbefda383cd68" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81587f0016414f1291b4e31c813f8a10" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "378055cb28e2485a84fc4225671fa425" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44765 } health_report { overall_health: UNKNOWN } } }
W20260807 11:31:50.133324 6019 tablet.cc:2369] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:31:50.163008 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:50.157109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.24.103.66:36271)
0807 11:31:50.157312 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:50.157316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:50.157433 (+ 117us) server_negotiation.cc:407] Connection header received
0807 11:31:50.157730 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:50.157735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:50.157833 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:50.157967 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:31:50.158390 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.159249 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.160797 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.161195 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.161332 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:50.161901 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:31:50.162398 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:31:50.162509 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:50.162627 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 11:31:50.162697 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260807 11:31:50.163314 5707 tablet.cc:2369] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:31:50.171942 5967 raft_consensus.cc:1261] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425 [term 1 FOLLOWER]: Refusing update from remote peer 36ac14d7cac94c2697afbefda383cd68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:31:50.171972 5652 raft_consensus.cc:1261] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10 [term 1 FOLLOWER]: Refusing update from remote peer 36ac14d7cac94c2697afbefda383cd68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:31:50.173074 6054 consensus_queue.cc:1035] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "378055cb28e2485a84fc4225671fa425" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:31:50.173478 6051 consensus_queue.cc:1035] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81587f0016414f1291b4e31c813f8a10" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:31:50.186393 6059 mvcc.cc:205] Tried to move back new op lower bound from 7315875062454218752 to 7315875062195924992. Current Snapshot: MvccSnapshot[applied={T|T < 7315875062454218752}]
I20260807 11:31:50.202927 5457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:31:50.203274 5457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:31:50.206892 5457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bae0ba0231284fefbc39ae581d110c18: Sending DeleteTablet for 3 replicas of tablet 90740e5fbd3a4e5aaf9a17aee811dd78
I20260807 11:31:50.208182 5791 tablet_service.cc:1514] Processing DeleteTablet for tablet 90740e5fbd3a4e5aaf9a17aee811dd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:31:50 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:47740
I20260807 11:31:50.208470 5628 tablet_service.cc:1514] Processing DeleteTablet for tablet 90740e5fbd3a4e5aaf9a17aee811dd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:31:50 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:36752
I20260807 11:31:50.208527 5943 tablet_service.cc:1514] Processing DeleteTablet for tablet 90740e5fbd3a4e5aaf9a17aee811dd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:31:50 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:33664
I20260807 11:31:50.211395 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 5548
I20260807 11:31:50.246924 6072 tablet_replica.cc:330] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: stopping tablet replica
I20260807 11:31:50.247925 6072 raft_consensus.cc:2229] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:31:50.252421 6071 tablet_replica.cc:330] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68: stopping tablet replica
I20260807 11:31:50.252985 6071 raft_consensus.cc:2229] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:31:50.251683 6072 raft_consensus.cc:2258] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:31:50.253986 6073 tablet_replica.cc:330] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: stopping tablet replica
I20260807 11:31:50.254416 6071 raft_consensus.cc:2258] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:31:50.254531 6073 raft_consensus.cc:2229] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:31:50.255131 6073 raft_consensus.cc:2258] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:31:50.256880 6071 ts_tablet_manager.cc:1905] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:31:50.256901 6072 ts_tablet_manager.cc:1905] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:31:50.259418 6073 ts_tablet_manager.cc:1905] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:31:50.264354 6072 ts_tablet_manager.cc:1918] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:31:50.264622 6072 log.cc:1196] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/90740e5fbd3a4e5aaf9a17aee811dd78
I20260807 11:31:50.264601 6071 ts_tablet_manager.cc:1918] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:31:50.264772 6071 log.cc:1196] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/90740e5fbd3a4e5aaf9a17aee811dd78
I20260807 11:31:50.265376 6072 ts_tablet_manager.cc:1939] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 81587f0016414f1291b4e31c813f8a10: Deleting consensus metadata
I20260807 11:31:50.265554 6071 ts_tablet_manager.cc:1939] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 36ac14d7cac94c2697afbefda383cd68: Deleting consensus metadata
I20260807 11:31:50.267565 5439 catalog_manager.cc:4841] TS 81587f0016414f1291b4e31c813f8a10 (127.24.103.65:45623): tablet 90740e5fbd3a4e5aaf9a17aee811dd78 (table test-table [id=ae802239d5b34e8fa275fc80f0496ff7]) successfully deleted
I20260807 11:31:50.267639 5442 catalog_manager.cc:4841] TS 36ac14d7cac94c2697afbefda383cd68 (127.24.103.66:36271): tablet 90740e5fbd3a4e5aaf9a17aee811dd78 (table test-table [id=ae802239d5b34e8fa275fc80f0496ff7]) successfully deleted
Aug 07 11:31:50 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:31:50.269433 6073 ts_tablet_manager.cc:1918] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:31:50.269659 6073 log.cc:1196] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/90740e5fbd3a4e5aaf9a17aee811dd78
I20260807 11:31:50.270244 6073 ts_tablet_manager.cc:1939] T 90740e5fbd3a4e5aaf9a17aee811dd78 P 378055cb28e2485a84fc4225671fa425: Deleting consensus metadata
I20260807 11:31:50.272356 5441 catalog_manager.cc:4841] TS 378055cb28e2485a84fc4225671fa425 (127.24.103.67:44765): tablet 90740e5fbd3a4e5aaf9a17aee811dd78 (table test-table [id=ae802239d5b34e8fa275fc80f0496ff7]) successfully deleted
I20260807 11:31:50.278259 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:50.213063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.24.103.65:45623)
0807 11:31:50.213233 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:50.213241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:50.242956 (+ 29715us) server_negotiation.cc:407] Connection header received
0807 11:31:50.243686 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:50.243707 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:50.243827 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:50.243922 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:31:50.248302 (+ 4380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.249863 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.253469 (+ 3606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.254160 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.254297 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:50.270534 (+ 16237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:31:50.270549 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:31:50.270565 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:31:50.270614 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:31:50.274593 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:50.275621 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:50.275631 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:50.275634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:50.275806 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:50.276262 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:50.276266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:50.276269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:50.276759 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:31:50.277060 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:50.277417 (+ 357us) server_negotiation.cc:299] Negotiation successful
0807 11:31:50.277673 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 11:31:50.278980 5696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:31:50.378252 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 5548
I20260807 11:31:50.406224 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 5711
Aug 07 11:31:50 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:31:50.424261 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:50.407071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.24.103.66:36271)
0807 11:31:50.407258 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:50.407263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:50.407324 (+ 61us) server_negotiation.cc:407] Connection header received
0807 11:31:50.407520 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:50.407524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:50.407592 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:50.407688 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:31:50.409356 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.410209 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.412363 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.412776 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.412873 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:50.418059 (+ 5186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:31:50.418071 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:31:50.418085 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:31:50.418132 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:31:50.421272 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:50.422013 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:50.422020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:50.422022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:50.422181 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:50.422587 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:50.422592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:50.422594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:50.422939 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:31:50.423308 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:50.423651 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 11:31:50.423879 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:31:50.424934 5859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:31:50.541822 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 5711
I20260807 11:31:50.564412 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 5872
Aug 07 11:31:50 dist-test-slave-wl2h krb5kdc[4919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:31:50.580340 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:31:50.565275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.24.103.67:44765)
0807 11:31:50.565462 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 11:31:50.565467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:31:50.565486 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:31:50.565573 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:31:50.565577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:31:50.565649 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:31:50.565732 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:31:50.567081 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.567826 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.569342 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:31:50.569720 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:31:50.569829 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:31:50.574533 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:31:50.574547 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:31:50.574562 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:31:50.574597 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:31:50.577483 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:50.578180 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:50.578187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:50.578189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:50.578325 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:31:50.578704 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:31:50.578709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:31:50.578711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:31:50.579183 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:31:50.579433 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 11:31:50.579726 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 11:31:50.579977 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:31:50.581032 6012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:31:50.673600 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 5872
I20260807 11:31:50.694430 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 5423
I20260807 11:31:50.695730 5493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:31:50.811990 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 5423
I20260807 11:31:50.839872 24989 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 11:31:51.186251 24989 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 11:31:51.186612 24989 mini_ranger.cc:66] Stopping Ranger...
I20260807 11:31:51.533334 24989 mini_ranger.cc:68] Stopped Ranger
2026-08-07 11:31:51.533 UTC [4952] LOG: received smart shutdown request
2026-08-07 11:31:51.538 UTC [4952] LOG: background worker "logical replication launcher" (PID 4961) exited with exit code 1
2026-08-07 11:31:51.538 UTC [4956] LOG: shutting down
2026-08-07 11:31:51.756 UTC [4952] LOG: database system is shut down
2026-08-07T11:31:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53608 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:31:52 dist-test-slave-wl2h krb5kdc[6089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:31:52 dist-test-slave-wl2h krb5kdc[6089](info): set up 2 sockets
Aug 07 11:31:52 dist-test-slave-wl2h krb5kdc[6089](info): commencing operation
krb5kdc: starting...
W20260807 11:31:53.137586 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:31:53 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:31:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:31:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/rangeradmin_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/rangeradmin_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/rangerlookup_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/rangerlookup_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
I20260807 11:31:53.567464 24989 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taska8aRx7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/postgres -l logfile start
2026-08-07 11:31:56.395 UTC [6121] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 11:31:56.395 UTC [6121] LOG: listening on IPv4 address "127.24.103.84", port 34403
2026-08-07 11:31:56.399 UTC [6121] LOG: listening on Unix socket "/tmp/.s.PGSQL.34403"
2026-08-07 11:31:56.424 UTC [6126] LOG: database system was shut down at 2026-08-07 11:31:54 UTC
2026-08-07 11:31:56.428 UTC [6121] LOG: database system is ready to accept connections
I20260807 11:31:57.505028 24989 mini_postgres.cc:96] Postgres bound to 34403
2026-08-07 11:31:57.530 UTC [6134] FATAL: database "slave" does not exist
127.24.103.84:34403 - accepting connections
I20260807 11:31:57.530790 24989 mini_ranger.cc:161] Starting Ranger...
I20260807 11:31:57.563941 24989 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 11:31:58.166553 24989 mini_ranger.cc:87] Created ranger Postgres database
I20260807 11:31:58.166704 24989 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-admin
2026-08-07 11:31:58,483 [I] DB FLAVOR :POSTGRES
2026-08-07 11:31:58,484 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 11:31:58,484 [I] Checking connection..
2026-08-07 11:31:58,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 11:31:58,747 [I] Checking connection passed.
2026-08-07 11:31:58,747 [I] --------- Verifying version history table ---------
2026-08-07 11:31:58,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:31:59,026 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 11:31:59,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:31:59,304 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 11:31:59,305 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 11:31:59,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 11:31:59.594 UTC [6206] WARNING: there is no transaction in progress
2026-08-07 11:31:59,602 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 11:31:59,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 11:31:59,885 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 11:31:59,885 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 11:31:59,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 11:32:00,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 11:32:00,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wl2h.c.gcp-upstream.internal','N') ;"
2026-08-07 11:32:00,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 11:32:00,960 [I] Table x_portal_user does not exist in database ranger
2026-08-07 11:32:00,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 11:32:01,245 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 11:32:01,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 11:32:01,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 11:32:01,814 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 11:32:01,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 11:32:02.933 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:02.957 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:02.978 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:02.997 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.023 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.050 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.118 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.129 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.149 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.161 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.172 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.189 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.199 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.207 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.216 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03.225 UTC [6328] WARNING: there is no transaction in progress
2026-08-07 11:32:03,994 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 11:32:03,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 11:32:04,253 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 11:32:04,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 11:32:04,505 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 11:32:04,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 11:32:04,765 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 11:32:04,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 11:32:05,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 11:32:05,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wl2h.c.gcp-upstream.internal';"
2026-08-07 11:32:05,567 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 11:32:05,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 11:32:05,822 [I] DB_PATCHES have already been applied
I20260807 11:32:05.829013 24989 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-admin:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taska8aRx7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 11:32:05.829126 24989 mini_ranger.cc:193] Using host: 127.24.103.84
I20260807 11:32:05.850301 24989 mini_ranger.cc:227] Ranger admin URL: http://127.24.103.84:56393
Aug 07, 2026 11:32:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 11:32:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 11:32:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 11:32:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 11:32:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 11:32:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 11:32:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 11:32:07.115839 24989 mini_ranger.cc:160] Time spent starting Ranger: real 9.585s user 0.007s sys 0.054s
Aug 07, 2026 11:32:28 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 11:32:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 07, 2026 11:32:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 11:32:28 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 11:32:29 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 11:32:29.859030 24989 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.103.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.103.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/rangerkms_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.103.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/rangerkms_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.103.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.103.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.103.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/HTTP_127.24.103.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 11:32:30 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 11:32:30.097669 24989 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 11:32:30.126729 24989 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 11:32:31.154294 24989 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 11:32:31.154428 24989 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-kms
I20260807 11:32:31.154480 24989 mini_ranger_kms.cc:227] Using postgres at 127.24.103.84:34403
2026-08-07 11:32:31,546 [I] DB FLAVOR :POSTGRES
2026-08-07 11:32:31,547 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 11:32:31,547 [I] Checking connection
2026-08-07 11:32:31,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 11:32:31,811 [I] connection success
2026-08-07 11:32:31,812 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 11:32:31,812 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 11:32:31,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 11:32:32,100 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 11:32:32,100 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 11:32:32,101 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 11:32:32,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.103.84:34403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 11:32:32,413 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 11:32:32.955546 24989 mini_ranger_kms.cc:310] Created kms service
I20260807 11:32:33.165329 24989 mini_ranger_kms.cc:326] Created kudu user
I20260807 11:32:33.217100 24989 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 11:32:33 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:32:33.655949 24989 mini_ranger_kms.cc:384] Added ranger policy
I20260807 11:32:33.656351 24989 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-kms:/tmp/dist-test-taska8aRx7/build/asan/bin/postgresql.jar:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taska8aRx7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 11:32:33.656406 24989 mini_ranger_kms.cc:240] Using host: 127.24.103.84
I20260807 11:32:33.678776 24989 mini_ranger_kms.cc:276] Ranger KMS PID: 6537
I20260807 11:32:33.678925 24989 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.103.84:46875
Aug 07, 2026 11:32:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 11:32:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 11:32:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 11:32:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 11:32:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 11:32:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 11:32:35 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786102355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:36.188679 24989 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.091s user 0.097s sys 0.328s
I20260807 11:32:36.188966 24989 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 11:32:36.189160 24989 mini_ranger_kms.cc:401] 127.24.103.84:46875/kms/v1/keys
Aug 07 11:32:36 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
Aug 07 11:32:36 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.103.84@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
Aug 07 11:32:39 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:32:39.319590 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38957
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38957
--ranger_config_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:46875/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:32:39.697187 6589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:39.697558 6589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:39.697762 6589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:39.706091 6589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:32:39.706228 6589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:39.706354 6589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:39.706471 6589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:32:39.706568 6589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:32:39.718130 6589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.103.84:46875/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38957
--ranger_config_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:39.719666 6589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:39.721103 6589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:39.733776 6598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:39.733822 6595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:39.733934 6596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:39.735430 6589 server_base.cc:1034] running on GCE node
I20260807 11:32:39.736302 6589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:39.737917 6589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:39.739135 6589 hybrid_clock.cc:648] HybridClock initialized: now 1786102359739096 us; error 41 us; skew 500 ppm
I20260807 11:32:39.739529 6589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:39 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102359, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:39.746261 6589 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:32:39.748294 6589 webserver.cc:460] Webserver started at http://127.24.103.126:36825/ using document root <none> and password file <none>
I20260807 11:32:39.749032 6589 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:39.749177 6589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:32:39.749599 6589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:32:39 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102359, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:32:39.802667 6589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "43df95c82e404daaa4be1af94c494a25"
format_stamp: "Formatted at 2026-08-07 11:32:39 on dist-test-slave-wl2h"
server_key: "23b3b3d8d91a8717f6f27df333785cf6"
server_key_iv: "59d2cfa12b38761faf47799655462fe3"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:39.803741 6589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "43df95c82e404daaa4be1af94c494a25"
format_stamp: "Formatted at 2026-08-07 11:32:39 on dist-test-slave-wl2h"
server_key: "23b3b3d8d91a8717f6f27df333785cf6"
server_key_iv: "59d2cfa12b38761faf47799655462fe3"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:39.809635 6589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:32:39.826884 6606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:39.828536 6589 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:32:39.828751 6589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "43df95c82e404daaa4be1af94c494a25"
format_stamp: "Formatted at 2026-08-07 11:32:39 on dist-test-slave-wl2h"
server_key: "23b3b3d8d91a8717f6f27df333785cf6"
server_key_iv: "59d2cfa12b38761faf47799655462fe3"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:39.828984 6589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:39.844964 6589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:39.849282 6589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:39.849622 6589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:39.871173 6589 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38957
I20260807 11:32:39.871198 6660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38957 every 8 connection(s)
I20260807 11:32:39.872692 6589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:32:39.874526 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 6589
I20260807 11:32:39.874782 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:32:39.875618 24989 external_mini_cluster.cc:1428] Setting key 099999f2f330ad3ddcd857d9195276dc
I20260807 11:32:39.877619 6663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:32:39.891846 6663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25: Bootstrap starting.
I20260807 11:32:39.896224 6663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25: Neither blocks nor log segments found. Creating new log.
I20260807 11:32:39.897372 6663 log.cc:824] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25: Log is configured to *not* fsync() on all Append() calls
I20260807 11:32:39.902233 6663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25: No bootstrap required, opened a new log
Aug 07 11:32:39 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:39.913257 6663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } }
I20260807 11:32:39.913683 6663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:32:39.913798 6663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43df95c82e404daaa4be1af94c494a25, State: Initialized, Role: FOLLOWER
I20260807 11:32:39.914669 6663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } }
I20260807 11:32:39.914944 6663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:32:39.915176 6663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:32:39.915413 6663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:32:39.918220 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:39.877935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.24.103.126:38957)
0807 11:32:39.891678 (+ 13743us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:39.891695 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:39.891752 (+ 57us) server_negotiation.cc:407] Connection header received
0807 11:32:39.894339 (+ 2587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:39.894370 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:39.894916 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:39.895417 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:39.897266 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:39.899591 (+ 2325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:39.901506 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:39.902602 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:39.902818 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:39.908282 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:39.908294 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:39.908318 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:39.908380 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:39.912704 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:39.913759 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:39.913769 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:39.913777 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:39.913995 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:39.914518 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:39.914523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:39.914526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:39.915364 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:39.915673 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:39.916259 (+ 586us) server_negotiation.cc:299] Negotiation successful
0807 11:32:39.916735 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13457,"thread_start_us":13121,"threads_started":1}
I20260807 11:32:39.918274 6663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } }
I20260807 11:32:39.918919 6663 leader_election.cc:302] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43df95c82e404daaa4be1af94c494a25; no voters:
I20260807 11:32:39.919700 6663 leader_election.cc:288] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:32:39.919850 6669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:32:39.923352 6669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 1 LEADER]: Becoming Leader. State: Replica: 43df95c82e404daaa4be1af94c494a25, State: Running, Role: LEADER
I20260807 11:32:39.924165 6669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } }
I20260807 11:32:39.925577 6663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:32:39.927152 6670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43df95c82e404daaa4be1af94c494a25. Latest consensus state: current_term: 1 leader_uuid: "43df95c82e404daaa4be1af94c494a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } } }
I20260807 11:32:39.928651 6670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [sys.catalog]: This master's current role is: LEADER
I20260807 11:32:39.928361 6671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43df95c82e404daaa4be1af94c494a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } } }
I20260807 11:32:39.929603 6671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [sys.catalog]: This master's current role is: LEADER
I20260807 11:32:39.942620 6674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:32:39.949220 6674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:32:39.968050 6674 catalog_manager.cc:1347] Generated new cluster ID: 8763bcffad2244e996cabef4ffd0f036
I20260807 11:32:39.968235 6674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:32:39.993325 6674 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:32:39.996100 6674 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:32:40.014401 6674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25: Generated new TSK 0
I20260807 11:32:40.016386 6674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:32:40.020468 6663 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:32:40.979955 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:46875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38957
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:32:41 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786102361, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 11:32:41.479296 6719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:41.479883 6719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:41.480178 6719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:41.498541 6719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:41.498824 6719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:41.499218 6719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:32:41.523602 6719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.103.84:46875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:41.526087 6719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:41.528308 6719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:41.571336 6732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:41.573246 6731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:41.575103 6719 server_base.cc:1034] running on GCE node
W20260807 11:32:41.575907 6734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:41.576787 6719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:41.580384 6719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:41.581666 6719 hybrid_clock.cc:648] HybridClock initialized: now 1786102361581595 us; error 72 us; skew 500 ppm
I20260807 11:32:41.582042 6719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:41 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102361, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:41.587392 6719 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:32:41.596004 6719 webserver.cc:460] Webserver started at http://127.24.103.65:35801/ using document root <none> and password file <none>
I20260807 11:32:41.597126 6719 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:41.597352 6719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:32:41.597795 6719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:32:41 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102361, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:32:41.640467 6719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "ccb549cfef4a44e0a3b71499f1108953"
format_stamp: "Formatted at 2026-08-07 11:32:41 on dist-test-slave-wl2h"
server_key: "1c0496259934a5252df90d81162fad7a"
server_key_iv: "5f8df98e04f82949cbd02103ee7fc108"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:41.641913 6719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "ccb549cfef4a44e0a3b71499f1108953"
format_stamp: "Formatted at 2026-08-07 11:32:41 on dist-test-slave-wl2h"
server_key: "1c0496259934a5252df90d81162fad7a"
server_key_iv: "5f8df98e04f82949cbd02103ee7fc108"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:41.661255 6719 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.014s sys 0.003s
I20260807 11:32:41.693645 6741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:41.695755 6719 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.004s sys 0.004s
I20260807 11:32:41.696029 6719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "ccb549cfef4a44e0a3b71499f1108953"
format_stamp: "Formatted at 2026-08-07 11:32:41 on dist-test-slave-wl2h"
server_key: "1c0496259934a5252df90d81162fad7a"
server_key_iv: "5f8df98e04f82949cbd02103ee7fc108"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:41.696362 6719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:41.760200 6719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:41.765995 6719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:41.766492 6719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:41.774080 6719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:32:41.782608 6719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:32:41.782820 6719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 11:32:41.783114 6719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:32:41.783262 6719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:41.881842 6719 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:34363
I20260807 11:32:41.881858 6869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:34363 every 8 connection(s)
I20260807 11:32:41.883832 6719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:32:41.885694 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 6719
I20260807 11:32:41.885908 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:32:41.886742 24989 external_mini_cluster.cc:1428] Setting key 362ebc0fb31e8f0f07d327ab3c058750
Aug 07 11:32:41 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102361, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:41.941615 6873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:41.891180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:47789 (local address 127.24.103.126:38957)
0807 11:32:41.913971 (+ 22791us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:41.913980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:41.917995 (+ 4015us) server_negotiation.cc:407] Connection header received
0807 11:32:41.918084 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:41.918089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:41.918210 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:41.918331 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:41.920809 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:41.922157 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:41.924831 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:41.925508 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:41.925625 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:41.931127 (+ 5502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:41.931135 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:41.931140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:41.931187 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:41.935430 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:41.937117 (+ 1687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:41.937124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:41.937131 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:41.937300 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:41.938102 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:41.938107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:41.938111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:41.938547 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:41.938737 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:41.940905 (+ 2168us) server_negotiation.cc:299] Negotiation successful
0807 11:32:41.941181 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22567,"thread_start_us":35890,"threads_started":1}
Aug 07 11:32:41 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102361, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:32:41.945034 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:41.890176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38957 (local address 127.24.103.65:47789)
0807 11:32:41.914736 (+ 24560us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:41.914816 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:41.917464 (+ 2648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:41.918857 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:41.918871 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:41.919632 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:41.920543 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:41.920573 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:41.922426 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:41.922431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:41.924620 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:41.924648 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:41.926032 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:41.926040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:41.926492 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:41.927734 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:41.927783 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:41.930770 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:41.936210 (+ 5440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:41.936221 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:41.936241 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:41.936932 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:41.937457 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:41.937462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:41.937466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:41.937928 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:41.938825 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:41.938834 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:41.939712 (+ 878us) client_negotiation.cc:769] Sending connection context
0807 11:32:41.940067 (+ 355us) client_negotiation.cc:240] Negotiation successful
0807 11:32:41.942758 (+ 2691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24128,"thread_start_us":37486,"threads_started":1}
I20260807 11:32:41.948262 6870 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38957
I20260807 11:32:41.948683 6870 heartbeater.cc:464] Registering TS with master...
I20260807 11:32:41.949636 6870 heartbeater.cc:511] Master 127.24.103.126:38957 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
I20260807 11:32:41.956176 6626 ts_manager.cc:194] Registered new tserver with Master: ccb549cfef4a44e0a3b71499f1108953 (127.24.103.65:34363)
I20260807 11:32:41.963465 6626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:47789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
W20260807 11:32:42.018803 24989 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.24.103.66: real 0.127s user 0.011s sys 0.015s
I20260807 11:32:42.019829 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:46875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38957
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:32:42 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102361, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
W20260807 11:32:42.551625 6876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:42.552162 6876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:42.552364 6876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:42.564328 6876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:42.564579 6876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:42.564850 6876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:32:42.579109 6876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.103.84:46875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:42.581212 6876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:42.583038 6876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:42.600764 6886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:42.607038 6887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:42.608289 6876 server_base.cc:1034] running on GCE node
W20260807 11:32:42.608443 6890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:42.609165 6876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:42.610360 6876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:42.611631 6876 hybrid_clock.cc:648] HybridClock initialized: now 1786102362611569 us; error 57 us; skew 500 ppm
I20260807 11:32:42.612008 6876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:42 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102362, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:42.616077 6876 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:32:42.639797 6876 webserver.cc:460] Webserver started at http://127.24.103.66:39861/ using document root <none> and password file <none>
I20260807 11:32:42.640749 6876 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:42.641037 6876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:32:42.641508 6876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:32:42 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102362, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:32:42.661681 6876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "ae721ac7f280489b99a961d9c2f6b90a"
format_stamp: "Formatted at 2026-08-07 11:32:42 on dist-test-slave-wl2h"
server_key: "e05f9dd3eb567b883e9597d738533a5b"
server_key_iv: "a4a8fa60cc31f7f9d80eec6fe836b06c"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:42.662902 6876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "ae721ac7f280489b99a961d9c2f6b90a"
format_stamp: "Formatted at 2026-08-07 11:32:42 on dist-test-slave-wl2h"
server_key: "e05f9dd3eb567b883e9597d738533a5b"
server_key_iv: "a4a8fa60cc31f7f9d80eec6fe836b06c"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:42.671260 6876 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260807 11:32:42.694832 6904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:42.696600 6876 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.000s
I20260807 11:32:42.696826 6876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "ae721ac7f280489b99a961d9c2f6b90a"
format_stamp: "Formatted at 2026-08-07 11:32:42 on dist-test-slave-wl2h"
server_key: "e05f9dd3eb567b883e9597d738533a5b"
server_key_iv: "a4a8fa60cc31f7f9d80eec6fe836b06c"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:42.697098 6876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:42.710469 6876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:42.714741 6876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:42.715152 6876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:42.716240 6876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:32:42.717777 6876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:32:42.717881 6876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:42.718050 6876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:32:42.718127 6876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:42.765225 6876 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:33287
I20260807 11:32:42.765241 7032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:33287 every 8 connection(s)
I20260807 11:32:42.766541 6876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:32:42.771927 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 6876
I20260807 11:32:42.772113 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:32:42.772647 24989 external_mini_cluster.cc:1428] Setting key ca75b7f9c17c51a214bfbdfd12791071
Aug 07 11:32:42 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102362, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:42.792253 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:42.770804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:59369 (local address 127.24.103.126:38957)
0807 11:32:42.771437 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:42.771443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:42.771469 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:32:42.773615 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:42.773621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:42.773700 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:42.773837 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:42.775676 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:42.776887 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:42.779208 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:42.779937 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:42.780098 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:42.784664 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:42.784671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:42.784676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:42.784733 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:42.788354 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:42.789175 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:42.789180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:42.789185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:42.789322 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:42.789808 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:42.789813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:42.789815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:42.790114 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:42.790280 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:42.791615 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0807 11:32:42.791835 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":988,"threads_started":1}
I20260807 11:32:42.793169 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:42.770261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38957 (local address 127.24.103.66:59369)
0807 11:32:42.771188 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:42.771240 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:42.773217 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:42.774093 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:42.774106 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:42.774696 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:42.775462 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:42.775486 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:42.777072 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:42.777078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:42.778999 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:42.779067 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:42.780181 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:42.780189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:42.780482 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:42.781335 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:42.781364 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:42.784356 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:42.788559 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:42.788569 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:42.788583 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:42.789020 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:42.789442 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:42.789448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:42.789453 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:42.789669 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:42.790251 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:42.790258 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:42.790656 (+ 398us) client_negotiation.cc:769] Sending connection context
0807 11:32:42.791093 (+ 437us) client_negotiation.cc:240] Negotiation successful
0807 11:32:42.791655 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":690,"threads_started":1}
I20260807 11:32:42.794728 7033 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38957
I20260807 11:32:42.795109 7033 heartbeater.cc:464] Registering TS with master...
I20260807 11:32:42.795864 7033 heartbeater.cc:511] Master 127.24.103.126:38957 requested a full tablet report, sending...
I20260807 11:32:42.797443 6626 ts_manager.cc:194] Registered new tserver with Master: ae721ac7f280489b99a961d9c2f6b90a (127.24.103.66:33287)
I20260807 11:32:42.798243 6626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:59369
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:32:42.852397 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:46875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38957
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:32:42.968366 6870 heartbeater.cc:503] Master 127.24.103.126:38957 was elected leader, sending a full tablet report...
W20260807 11:32:43.236770 7043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:43.237106 7043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:43.237296 7043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:43.244911 7043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:43.245080 7043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:43.245319 7043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:32:43.255640 7043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.103.84:46875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:43.257068 7043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:43.258507 7043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:43.272681 7050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:43.272686 7052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:43.272682 7049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:43.276892 7043 server_base.cc:1034] running on GCE node
I20260807 11:32:43.277603 7043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:43.278663 7043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:43.279917 7043 hybrid_clock.cc:648] HybridClock initialized: now 1786102363279845 us; error 63 us; skew 500 ppm
I20260807 11:32:43.280285 7043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:43 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102363, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:43.283984 7043 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:32:43.285825 7043 webserver.cc:460] Webserver started at http://127.24.103.67:35169/ using document root <none> and password file <none>
I20260807 11:32:43.286583 7043 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:43.286696 7043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:32:43.286988 7043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 11:32:43 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102363, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:32:43.304364 7043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "a328d62d8cf743f5a7976113523df55f"
format_stamp: "Formatted at 2026-08-07 11:32:43 on dist-test-slave-wl2h"
server_key: "881448bf5d2bd75fe6714c69f037aa92"
server_key_iv: "2fa741e90701fc6db2b8e854f2a28f84"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:43.305393 7043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "a328d62d8cf743f5a7976113523df55f"
format_stamp: "Formatted at 2026-08-07 11:32:43 on dist-test-slave-wl2h"
server_key: "881448bf5d2bd75fe6714c69f037aa92"
server_key_iv: "2fa741e90701fc6db2b8e854f2a28f84"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:43.311939 7043 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260807 11:32:43.331845 7060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:43.336613 7043 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.002s
I20260807 11:32:43.336866 7043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "a328d62d8cf743f5a7976113523df55f"
format_stamp: "Formatted at 2026-08-07 11:32:43 on dist-test-slave-wl2h"
server_key: "881448bf5d2bd75fe6714c69f037aa92"
server_key_iv: "2fa741e90701fc6db2b8e854f2a28f84"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:43.337184 7043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:43.381662 7043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:43.386974 7043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:43.387373 7043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:43.388406 7043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:32:43.389937 7043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:32:43.390074 7043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:43.390230 7043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:32:43.390305 7043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:43.428748 7043 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:43937
I20260807 11:32:43.428797 7191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:43937 every 8 connection(s)
I20260807 11:32:43.430228 7043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:32:43.440423 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 7043
I20260807 11:32:43.440670 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:32:43.441226 24989 external_mini_cluster.cc:1428] Setting key a23e62957701fd75cc5b6643da1d80b8
Aug 07 11:32:43 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102363, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:43.451988 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:43.433450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:44205 (local address 127.24.103.126:38957)
0807 11:32:43.434024 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:43.434034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:43.435085 (+ 1051us) server_negotiation.cc:407] Connection header received
0807 11:32:43.436896 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:43.436900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:43.436990 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:43.437105 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:43.438659 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:43.439524 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:43.441397 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:43.441860 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:43.442013 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:43.445796 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:43.445801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:43.445807 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:43.445846 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:43.448724 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:43.449518 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:43.449525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:43.449529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:43.449664 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:43.450132 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:43.450138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:43.450141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:43.450425 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:43.450594 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:43.451379 (+ 785us) server_negotiation.cc:299] Negotiation successful
0807 11:32:43.451609 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":275,"threads_started":1}
I20260807 11:32:43.452970 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:43.434098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38957 (local address 127.24.103.67:44205)
0807 11:32:43.434894 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:43.434933 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:43.436615 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:43.437372 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:43.437381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:43.437847 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:43.438428 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:43.438450 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:43.439720 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:43.439724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:43.441199 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:43.441212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:43.442121 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:43.442132 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:43.442404 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:43.443168 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:43.443192 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:43.445523 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:43.448876 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:43.448899 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:43.448921 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:43.449394 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:43.449779 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:43.449784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:43.449788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:43.450014 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:43.450560 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:43.450569 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:43.450937 (+ 368us) client_negotiation.cc:769] Sending connection context
0807 11:32:43.451289 (+ 352us) client_negotiation.cc:240] Negotiation successful
0807 11:32:43.451838 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":346,"threads_started":1}
I20260807 11:32:43.454522 7192 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38957
I20260807 11:32:43.454859 7192 heartbeater.cc:464] Registering TS with master...
I20260807 11:32:43.455783 7192 heartbeater.cc:511] Master 127.24.103.126:38957 requested a full tablet report, sending...
I20260807 11:32:43.457479 6626 ts_manager.cc:194] Registered new tserver with Master: a328d62d8cf743f5a7976113523df55f (127.24.103.67:43937)
I20260807 11:32:43.458312 6626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:44205
I20260807 11:32:43.469817 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:32:43.471923 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 6719
Aug 07 11:32:43 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:32:43.490962 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:43.473187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.24.103.65:34363)
0807 11:32:43.473828 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:43.473835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:43.473864 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:32:43.473926 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:43.473933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:43.474163 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:43.474304 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:43.475926 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:43.477118 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:43.478655 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:43.479339 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:43.479459 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:43.484217 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:43.484231 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:43.484247 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:43.484290 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:43.487753 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:43.488592 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:43.488600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:43.488603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:43.488785 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:43.489274 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:43.489278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:43.489280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:43.489658 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:43.489927 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:43.490283 (+ 356us) server_negotiation.cc:299] Negotiation successful
0807 11:32:43.490531 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":307,"threads_started":1}
I20260807 11:32:43.492707 6864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:43.579242 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 6719
I20260807 11:32:43.600328 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 6876
Aug 07 11:32:43 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:32:43.623133 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:43.601393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.24.103.66:33287)
0807 11:32:43.601885 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:43.601891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:43.601916 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:32:43.601977 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:43.601983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:43.602188 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:43.602375 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:43.603727 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:43.604702 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:43.606080 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:43.606741 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:43.606890 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:43.616512 (+ 9622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:43.616527 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:43.616543 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:43.616587 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:43.619846 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:43.620720 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:43.620729 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:43.620731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:43.620880 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:43.621300 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:43.621307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:43.621310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:43.621698 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:43.622049 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:43.622376 (+ 327us) server_negotiation.cc:299] Negotiation successful
0807 11:32:43.622660 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":228,"threads_started":1}
I20260807 11:32:43.624485 7027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:43.710546 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 6876
I20260807 11:32:43.731329 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 7043
Aug 07 11:32:43 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:32:43.748510 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:43.732385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.24.103.67:43937)
0807 11:32:43.733090 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:43.733097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:43.733130 (+ 33us) server_negotiation.cc:407] Connection header received
0807 11:32:43.733193 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:43.733199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:43.733422 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:43.733552 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:43.734943 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:43.736184 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:43.737473 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:43.738016 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:43.738165 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:43.742565 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:43.742576 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:43.742591 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:43.742624 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:43.745770 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:43.746518 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:43.746524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:43.746527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:43.746684 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:43.747061 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:43.747072 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:43.747074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:43.747403 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:43.747695 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:43.747886 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 11:32:43.748084 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":286,"threads_started":1}
I20260807 11:32:43.749866 7187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:43.834326 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 7043
I20260807 11:32:43.854173 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 6589
I20260807 11:32:43.855463 6656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:43.964658 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 6589
I20260807 11:32:43.992198 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38957
--webserver_interface=127.24.103.126
--webserver_port=36825
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38957
--ranger_config_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:46875/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:46875/kms with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:32:44.329360 7204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:44.329727 7204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:44.329887 7204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:44.337477 7204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:32:44.337634 7204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:44.337754 7204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:44.337847 7204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:32:44.337939 7204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:32:44.348140 7204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:46875/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38957
--ranger_config_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:44.349445 7204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:44.350780 7204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:44.363206 7214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:44.363335 7211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:44.363314 7212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:44.363723 7204 server_base.cc:1034] running on GCE node
I20260807 11:32:44.364755 7204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:44.366150 7204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:44.367391 7204 hybrid_clock.cc:648] HybridClock initialized: now 1786102364367334 us; error 51 us; skew 500 ppm
I20260807 11:32:44.367812 7204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:44 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102364, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:44.370949 7204 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:32:44.372427 7204 webserver.cc:460] Webserver started at http://127.24.103.126:36825/ using document root <none> and password file <none>
I20260807 11:32:44.373070 7204 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:44.373214 7204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 11:32:44 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102364, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:32:44.432338 7204 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 11:32:44.436097 7223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:44.437847 7204 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260807 11:32:44.438103 7204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "43df95c82e404daaa4be1af94c494a25"
format_stamp: "Formatted at 2026-08-07 11:32:39 on dist-test-slave-wl2h"
server_key: "23b3b3d8d91a8717f6f27df333785cf6"
server_key_iv: "59d2cfa12b38761faf47799655462fe3"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:44.439500 7204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:44.461974 7204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:44.466360 7204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:44.466651 7204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:44.488400 7204 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38957
I20260807 11:32:44.488430 7280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38957 every 8 connection(s)
I20260807 11:32:44.490110 7204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:32:44.492597 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 7204
I20260807 11:32:44.493861 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:46875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:34363
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:38957
--webserver_port=35801
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:46875/kms with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:32:44.500833 7282 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:32:44.506422 7282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25: Bootstrap starting.
I20260807 11:32:44.527639 7282 log.cc:824] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25: Log is configured to *not* fsync() on all Append() calls
I20260807 11:32:44.537173 7282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:32:44.537756 7282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25: Bootstrap complete.
I20260807 11:32:44.547789 7282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } }
I20260807 11:32:44.548694 7282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43df95c82e404daaa4be1af94c494a25, State: Initialized, Role: FOLLOWER
I20260807 11:32:44.549501 7282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } }
I20260807 11:32:44.549758 7282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:32:44.549883 7282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:32:44.550081 7282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:32:44.554335 7282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } }
I20260807 11:32:44.554912 7282 leader_election.cc:302] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43df95c82e404daaa4be1af94c494a25; no voters:
I20260807 11:32:44.555635 7282 leader_election.cc:288] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 11:32:44.555792 7286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:32:44.556962 7286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [term 2 LEADER]: Becoming Leader. State: Replica: 43df95c82e404daaa4be1af94c494a25, State: Running, Role: LEADER
I20260807 11:32:44.557693 7286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } }
I20260807 11:32:44.558027 7282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:32:44.560565 7288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43df95c82e404daaa4be1af94c494a25. Latest consensus state: current_term: 2 leader_uuid: "43df95c82e404daaa4be1af94c494a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } } }
I20260807 11:32:44.560590 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "43df95c82e404daaa4be1af94c494a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43df95c82e404daaa4be1af94c494a25" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38957 } } }
I20260807 11:32:44.560969 7288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [sys.catalog]: This master's current role is: LEADER
I20260807 11:32:44.561053 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25 [sys.catalog]: This master's current role is: LEADER
I20260807 11:32:44.568429 7296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:32:44.574239 7296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:32:44.575724 7296 catalog_manager.cc:1259] Loaded cluster ID: 8763bcffad2244e996cabef4ffd0f036
I20260807 11:32:44.575846 7296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:32:44.580314 7296 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:32:44.581908 7296 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25: Loaded TSK: 0
I20260807 11:32:44.583586 7296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:32:44.586334 7282 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260807 11:32:44.946725 7284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:44.947145 7284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:44.947326 7284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:44.959177 7284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:44.959363 7284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:44.959573 7284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:32:44.975648 7284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:46875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:34363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=35801
--tserver_master_addrs=127.24.103.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:44.977484 7284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:44.979403 7284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:45.010641 7322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:45.011240 7323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:45.014510 7284 server_base.cc:1034] running on GCE node
W20260807 11:32:45.015229 7325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:45.016490 7284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:45.018023 7284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:45.019328 7284 hybrid_clock.cc:648] HybridClock initialized: now 1786102365019250 us; error 77 us; skew 500 ppm
I20260807 11:32:45.019961 7284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:45 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102365, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:45.029486 7284 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:32:45.040004 7284 webserver.cc:460] Webserver started at http://127.24.103.65:35801/ using document root <none> and password file <none>
I20260807 11:32:45.041167 7284 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:45.041375 7284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 11:32:45 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102365, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:32:45.099000 7284 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.003s sys 0.008s
I20260807 11:32:45.108975 7335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:45.111037 7284 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.007s
I20260807 11:32:45.111382 7284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "ccb549cfef4a44e0a3b71499f1108953"
format_stamp: "Formatted at 2026-08-07 11:32:41 on dist-test-slave-wl2h"
server_key: "1c0496259934a5252df90d81162fad7a"
server_key_iv: "5f8df98e04f82949cbd02103ee7fc108"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:45.113059 7284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:45.149821 7284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:45.156564 7284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:45.157009 7284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:45.165172 7284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:32:45.173810 7284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:32:45.174046 7284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:45.174253 7284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:32:45.174399 7284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:45.253376 7284 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:34363
I20260807 11:32:45.253430 7462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:34363 every 8 connection(s)
I20260807 11:32:45.255225 7284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:32:45.257254 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 7284
I20260807 11:32:45.258200 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:46875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:33287
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:38957
--webserver_port=39861
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:46875/kms with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:32:45 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102365, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:45.304528 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:45.259114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38957 (local address 127.24.103.65:40305)
0807 11:32:45.276404 (+ 17290us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:45.276462 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:45.278435 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:45.281382 (+ 2947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:45.281393 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:45.282073 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:45.282906 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:45.282934 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:45.285485 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:45.285491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:45.287406 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:45.287420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:45.289196 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:45.289209 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:45.289566 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:45.291060 (+ 1494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:45.291109 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:45.294370 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:45.299284 (+ 4914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:45.299294 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:45.299311 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:45.299902 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:45.300466 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:45.300472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:45.300476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:45.300736 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:45.301754 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:45.301763 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:45.302162 (+ 399us) client_negotiation.cc:769] Sending connection context
0807 11:32:45.302483 (+ 321us) client_negotiation.cc:240] Negotiation successful
0807 11:32:45.303163 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16977,"thread_start_us":16732,"threads_started":1}
I20260807 11:32:45.304543 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:45.259118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:40305 (local address 127.24.103.126:38957)
0807 11:32:45.275730 (+ 16612us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:45.275754 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:45.278693 (+ 2939us) server_negotiation.cc:407] Connection header received
0807 11:32:45.279586 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:45.279603 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:45.280104 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:45.280509 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:45.283244 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:45.285285 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:45.287581 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:45.288626 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:45.288806 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:45.294778 (+ 5972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:45.294792 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:45.294812 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:45.294865 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:45.299105 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:45.300095 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:45.300105 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:45.300114 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:45.300329 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:45.300879 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:45.300884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:45.300887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:45.301608 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:45.301830 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:45.302763 (+ 933us) server_negotiation.cc:299] Negotiation successful
0807 11:32:45.303227 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16252,"thread_start_us":21645,"threads_started":1}
I20260807 11:32:45.308899 7463 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38957
I20260807 11:32:45.309242 7463 heartbeater.cc:464] Registering TS with master...
I20260807 11:32:45.310127 7463 heartbeater.cc:511] Master 127.24.103.126:38957 requested a full tablet report, sending...
W20260807 11:32:45.313464 7463 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:38957 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 11:32:45.313697 7463 heartbeater.cc:464] Registering TS with master...
I20260807 11:32:45.313985 7463 heartbeater.cc:511] Master 127.24.103.126:38957 requested a full tablet report, sending...
I20260807 11:32:45.315306 7463 heartbeater.cc:464] Registering TS with master...
I20260807 11:32:45.315657 7463 heartbeater.cc:511] Master 127.24.103.126:38957 requested a full tablet report, sending...
W20260807 11:32:45.320024 7463 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 11:32:45.723873 7467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:45.724160 7467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:45.724324 7467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:45.732292 7467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:45.732465 7467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:45.732653 7467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:32:45.743453 7467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:46875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:33287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=39861
--tserver_master_addrs=127.24.103.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:45.744861 7467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:45.746359 7467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:45.763614 7486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:45.763639 7484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:45.765508 7483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:45.765803 7467 server_base.cc:1034] running on GCE node
I20260807 11:32:45.766736 7467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:45.768343 7467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:45.769685 7467 hybrid_clock.cc:648] HybridClock initialized: now 1786102365769593 us; error 88 us; skew 500 ppm
I20260807 11:32:45.770097 7467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:45 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102365, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:45.774401 7467 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:32:45.776511 7467 webserver.cc:460] Webserver started at http://127.24.103.66:39861/ using document root <none> and password file <none>
I20260807 11:32:45.777529 7467 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:45.777717 7467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 11:32:45 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102365, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:32:45.804373 7467 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:32:45.808733 7494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:45.810601 7467 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:32:45.810878 7467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "ae721ac7f280489b99a961d9c2f6b90a"
format_stamp: "Formatted at 2026-08-07 11:32:42 on dist-test-slave-wl2h"
server_key: "e05f9dd3eb567b883e9597d738533a5b"
server_key_iv: "a4a8fa60cc31f7f9d80eec6fe836b06c"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:45.812520 7467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:45.860998 7467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:45.866652 7467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:45.867133 7467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:45.875070 7467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:32:45.883915 7467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:32:45.884135 7467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 11:32:45.884312 7467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:32:45.884433 7467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 11:32:45 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786102365, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:46.091226 7467 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:33287
I20260807 11:32:46.091274 7627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:33287 every 8 connection(s)
I20260807 11:32:46.093420 7467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:32:46.094070 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 7467
I20260807 11:32:46.095129 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.103.84:46875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:43937
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:38957
--webserver_port=35169
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:46875/kms with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:32:46 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102365, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:46.129626 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.100502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:49687 (local address 127.24.103.126:38957)
0807 11:32:46.106344 (+ 5842us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:46.106353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:46.106378 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:32:46.106452 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:46.106460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:46.106565 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:46.106687 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:46.110278 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.111469 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.114214 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.114957 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.115122 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:46.120462 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:46.120473 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:46.120481 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:46.120559 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:46.125011 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:46.126208 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:46.126216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:46.126223 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:46.126390 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:46.126996 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:46.127004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:46.127009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:46.127459 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:46.127598 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:46.128781 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0807 11:32:46.129073 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5663,"thread_start_us":10508,"threads_started":1}
I20260807 11:32:46.132756 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.099671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38957 (local address 127.24.103.66:49687)
0807 11:32:46.102172 (+ 2501us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:46.102237 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:46.104951 (+ 2714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:46.108417 (+ 3466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:46.108432 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:46.109143 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:46.110024 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:46.110052 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.111698 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.111707 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:46.113974 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:46.114014 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.115521 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.115531 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:46.115867 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:46.116843 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:46.116880 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:46.120036 (+ 3156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:46.125251 (+ 5215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:46.125263 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:46.125287 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:46.126029 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:46.126546 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:46.126553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:46.126560 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:46.126856 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:46.127721 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:46.127732 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:46.128258 (+ 526us) client_negotiation.cc:769] Sending connection context
0807 11:32:46.130290 (+ 2032us) client_negotiation.cc:240] Negotiation successful
0807 11:32:46.131002 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2181,"thread_start_us":1918,"threads_started":1}
I20260807 11:32:46.135227 7628 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38957
I20260807 11:32:46.135673 7628 heartbeater.cc:464] Registering TS with master...
I20260807 11:32:46.136756 7628 heartbeater.cc:511] Master 127.24.103.126:38957 requested a full tablet report, sending...
I20260807 11:32:46.139168 7243 ts_manager.cc:194] Registered new tserver with Master: ae721ac7f280489b99a961d9c2f6b90a (127.24.103.66:33287)
I20260807 11:32:46.141433 7243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:49687
I20260807 11:32:46.320176 7463 heartbeater.cc:464] Registering TS with master...
I20260807 11:32:46.320734 7463 heartbeater.cc:511] Master 127.24.103.126:38957 requested a full tablet report, sending...
I20260807 11:32:46.322310 7243 ts_manager.cc:194] Registered new tserver with Master: ccb549cfef4a44e0a3b71499f1108953 (127.24.103.65:34363)
I20260807 11:32:46.323596 7243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:40305
Aug 07 11:32:46 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102365, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
W20260807 11:32:46.590292 7635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:46.590821 7635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:46.591001 7635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:46.602975 7635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:46.603233 7635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:46.603452 7635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:32:46.619675 7635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.103.84:46875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:43937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=35169
--tserver_master_addrs=127.24.103.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:46.621644 7635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:46.623471 7635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:46.653782 7640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:46.655843 7641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:46.664736 7635 server_base.cc:1034] running on GCE node
W20260807 11:32:46.665112 7643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:46.666023 7635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:46.667397 7635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:46.668700 7635 hybrid_clock.cc:648] HybridClock initialized: now 1786102366668634 us; error 52 us; skew 500 ppm
I20260807 11:32:46.669178 7635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:46 dist-test-slave-wl2h krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102366, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:46.673892 7635 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:32:46.676277 7635 webserver.cc:460] Webserver started at http://127.24.103.67:35169/ using document root <none> and password file <none>
I20260807 11:32:46.677314 7635 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:46.677490 7635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 11:32:46 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102366, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:32:46.717366 7635 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.003s
I20260807 11:32:46.721398 7657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:46.723357 7635 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:32:46.723551 7635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "a328d62d8cf743f5a7976113523df55f"
format_stamp: "Formatted at 2026-08-07 11:32:43 on dist-test-slave-wl2h"
server_key: "881448bf5d2bd75fe6714c69f037aa92"
server_key_iv: "2fa741e90701fc6db2b8e854f2a28f84"
server_key_version: "kuduclusterkey@0"
I20260807 11:32:46.725055 7635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:46.744007 7635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:46.749709 7635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:46.750058 7635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:46.751540 7635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:32:46.753427 7635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:32:46.753558 7635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:46.753695 7635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:32:46.753790 7635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:46.809087 7635 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:43937
I20260807 11:32:46.809156 7782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:43937 every 8 connection(s)
I20260807 11:32:46.812165 7635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:32:46.817729 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 7635
Aug 07 11:32:46 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102366, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:46.857733 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.819348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.24.103.126:38957)
0807 11:32:46.829515 (+ 10167us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:46.829520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:46.829535 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:32:46.829590 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:46.829594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:46.829742 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:46.829906 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:46.832102 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.833107 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.835060 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.835506 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.835727 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:46.838540 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:46.838547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:46.838557 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:46.838597 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:46.844588 (+ 5991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:46.847826 (+ 3238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:46.847833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:46.847837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:46.848010 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:46.849555 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:46.849561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:46.849563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:46.849949 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:46.850233 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:46.857052 (+ 6819us) server_negotiation.cc:299] Negotiation successful
0807 11:32:46.857312 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10079,"thread_start_us":9956,"threads_started":1}
I20260807 11:32:46.862301 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.816335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38957 (local address 127.24.103.67:50813)
0807 11:32:46.823879 (+ 7544us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:46.823951 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:46.839637 (+ 15686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:46.840697 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:46.840707 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:46.841165 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:46.841904 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:46.841937 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.844083 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.844092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:46.845910 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:46.845924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.847280 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.847285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:46.847479 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:46.848327 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:46.848367 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:46.850960 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:46.855315 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:46.855325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:46.855344 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:46.856019 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:46.856500 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:46.856506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:46.856509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:46.856829 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:46.859539 (+ 2710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:46.859546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:46.859953 (+ 407us) client_negotiation.cc:769] Sending connection context
0807 11:32:46.860325 (+ 372us) client_negotiation.cc:240] Negotiation successful
0807 11:32:46.860946 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7058,"thread_start_us":7174,"threads_started":1}
I20260807 11:32:46.863454 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.817017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:50813 (local address 127.24.103.126:38957)
0807 11:32:46.828454 (+ 11437us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:46.828463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:46.828493 (+ 30us) server_negotiation.cc:407] Connection header received
0807 11:32:46.840005 (+ 11512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:46.840010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:46.840101 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:46.840213 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:46.842138 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.843882 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.846061 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.846815 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.846951 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:46.851411 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:46.851419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:46.851423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:46.851464 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:46.855079 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:46.856192 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:46.856198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:46.856201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:46.856354 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:46.858433 (+ 2079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:46.858441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:46.858447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:46.859297 (+ 850us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:46.859450 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:46.862803 (+ 3353us) server_negotiation.cc:299] Negotiation successful
0807 11:32:46.863096 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11268,"thread_start_us":11111,"threads_started":1}
I20260807 11:32:46.865588 7783 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38957
I20260807 11:32:46.865963 7783 heartbeater.cc:464] Registering TS with master...
I20260807 11:32:46.866806 7783 heartbeater.cc:511] Master 127.24.103.126:38957 requested a full tablet report, sending...
I20260807 11:32:46.868673 7243 ts_manager.cc:194] Registered new tserver with Master: a328d62d8cf743f5a7976113523df55f (127.24.103.67:43937)
I20260807 11:32:46.869796 7243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:50813
I20260807 11:32:46.874540 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:32:46.898063 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.878583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35386 (local address 127.24.103.126:38957)
0807 11:32:46.879798 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:46.879804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:46.880357 (+ 553us) server_negotiation.cc:407] Connection header received
0807 11:32:46.880543 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:46.880549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:46.880639 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:46.880751 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:46.882937 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.884046 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.886270 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:46.886931 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:46.887077 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:46.890676 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:46.890682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:46.890688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:46.890734 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:46.894490 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:46.895516 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:46.895522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:46.895526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:46.895686 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:46.896166 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:46.896171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:46.896174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:46.896576 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:46.896795 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:46.897458 (+ 663us) server_negotiation.cc:299] Negotiation successful
0807 11:32:46.897707 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1044}
Aug 07 11:32:46 dist-test-slave-wl2h krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102365, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for HTTP/127.24.103.84@KRBTEST.COM
I20260807 11:32:46.905014 7243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:32:46.923484 7243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:32:47.031351 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.960182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:33287 (local address 127.0.0.1:34034)
0807 11:32:47.012846 (+ 52664us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.012913 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:47.013211 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:47.020454 (+ 7243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:47.020460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:47.020485 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:32:47.020875 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:47.020891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.023452 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.023459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:47.027568 (+ 4109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:47.027587 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.030190 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.030198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:47.030435 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.030497 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 11:32:47.030646 (+ 149us) client_negotiation.cc:240] Negotiation successful
0807 11:32:47.030875 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52264,"thread_start_us":62895,"threads_started":1}
I20260807 11:32:47.034555 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.960807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.24.103.66:33287)
0807 11:32:47.017190 (+ 56383us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:47.017202 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:47.017235 (+ 33us) server_negotiation.cc:407] Connection header received
0807 11:32:47.017354 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:47.017363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:47.017651 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:47.017914 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:32:47.021118 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.022931 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.027816 (+ 4885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.029589 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.029760 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.029991 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:47.033902 (+ 3911us) server_negotiation.cc:299] Negotiation successful
0807 11:32:47.034128 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56179,"thread_start_us":57330,"threads_started":1}
I20260807 11:32:47.039559 7552 tablet_service.cc:1467] Processing CreateTablet for tablet f4ec726b11044e62932fc7a56cfdb998 (DEFAULT_TABLE table=test-table [id=9006bc92d7cc4c8da48081195c01f4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:32:47.042944 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.980899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.24.103.65:34363)
0807 11:32:47.001239 (+ 20340us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:47.001250 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:47.014076 (+ 12826us) server_negotiation.cc:407] Connection header received
0807 11:32:47.014285 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:47.014294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:47.014609 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:47.014851 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:32:47.019095 (+ 4244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.020842 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.039576 (+ 18734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.041182 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.041327 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.041580 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:47.042346 (+ 766us) server_negotiation.cc:299] Negotiation successful
0807 11:32:47.042550 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20019,"thread_start_us":20897,"threads_started":1}
I20260807 11:32:47.043813 7552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ec726b11044e62932fc7a56cfdb998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:32:47.044027 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.980537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34363 (local address 127.0.0.1:48900)
0807 11:32:47.013914 (+ 33377us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.013936 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:47.014095 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:47.018240 (+ 4145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:47.018250 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:47.018291 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:32:47.018800 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:47.018822 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.022135 (+ 3313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.022140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:47.024710 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:47.024727 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.041853 (+ 17126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.041861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:47.042077 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.042117 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 11:32:47.043502 (+ 1385us) client_negotiation.cc:240] Negotiation successful
0807 11:32:47.043676 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33235,"thread_start_us":33675,"threads_started":1}
I20260807 11:32:47.048453 7390 tablet_service.cc:1467] Processing CreateTablet for tablet f4ec726b11044e62932fc7a56cfdb998 (DEFAULT_TABLE table=test-table [id=9006bc92d7cc4c8da48081195c01f4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:32:47.050210 7390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ec726b11044e62932fc7a56cfdb998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:32:47.054145 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.984679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:43937 (local address 127.0.0.1:50962)
0807 11:32:47.020122 (+ 35443us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.020144 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:47.020308 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:47.024897 (+ 4589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:47.024901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:47.024931 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:32:47.025389 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:47.025404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.037186 (+ 11782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.037203 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:47.039298 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:47.039319 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.053272 (+ 13953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.053280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:47.053480 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.053515 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:32:47.053617 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 11:32:47.053781 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35306,"thread_start_us":30344,"threads_started":1}
I20260807 11:32:47.055083 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:46.985130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.24.103.67:43937)
0807 11:32:47.001773 (+ 16643us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:47.001781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:47.021089 (+ 19308us) server_negotiation.cc:407] Connection header received
0807 11:32:47.021229 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:47.021238 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:47.021515 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:47.021750 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:32:47.035173 (+ 13423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.036945 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.051036 (+ 14091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.052649 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.052814 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.053059 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:47.054534 (+ 1475us) server_negotiation.cc:299] Negotiation successful
0807 11:32:47.054723 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16449,"thread_start_us":21893,"threads_started":1}
I20260807 11:32:47.059580 7710 tablet_service.cc:1467] Processing CreateTablet for tablet f4ec726b11044e62932fc7a56cfdb998 (DEFAULT_TABLE table=test-table [id=9006bc92d7cc4c8da48081195c01f4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:32:47.061285 7710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ec726b11044e62932fc7a56cfdb998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:32:47.097654 7814 tablet_bootstrap.cc:492] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: Bootstrap starting.
I20260807 11:32:47.101030 7816 tablet_bootstrap.cc:492] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953: Bootstrap starting.
I20260807 11:32:47.104938 7816 tablet_bootstrap.cc:654] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953: Neither blocks nor log segments found. Creating new log.
I20260807 11:32:47.105334 7815 tablet_bootstrap.cc:492] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: Bootstrap starting.
I20260807 11:32:47.105554 7814 tablet_bootstrap.cc:654] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: Neither blocks nor log segments found. Creating new log.
I20260807 11:32:47.106431 7816 log.cc:824] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953: Log is configured to *not* fsync() on all Append() calls
I20260807 11:32:47.107060 7814 log.cc:824] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: Log is configured to *not* fsync() on all Append() calls
I20260807 11:32:47.114648 7815 tablet_bootstrap.cc:654] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: Neither blocks nor log segments found. Creating new log.
I20260807 11:32:47.116112 7815 log.cc:824] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: Log is configured to *not* fsync() on all Append() calls
I20260807 11:32:47.116290 7816 tablet_bootstrap.cc:492] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953: No bootstrap required, opened a new log
I20260807 11:32:47.116680 7816 ts_tablet_manager.cc:1397] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20260807 11:32:47.122642 7815 tablet_bootstrap.cc:492] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: No bootstrap required, opened a new log
I20260807 11:32:47.122929 7815 ts_tablet_manager.cc:1397] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20260807 11:32:47.126554 7814 tablet_bootstrap.cc:492] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: No bootstrap required, opened a new log
I20260807 11:32:47.126871 7814 ts_tablet_manager.cc:1397] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.003s
I20260807 11:32:47.128255 7816 raft_consensus.cc:348] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae721ac7f280489b99a961d9c2f6b90a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33287 } } peers { permanent_uuid: "ccb549cfef4a44e0a3b71499f1108953" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34363 } } peers { permanent_uuid: "a328d62d8cf743f5a7976113523df55f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43937 } }
I20260807 11:32:47.128880 7816 raft_consensus.cc:374] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:32:47.129036 7816 raft_consensus.cc:729] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb549cfef4a44e0a3b71499f1108953, State: Initialized, Role: FOLLOWER
I20260807 11:32:47.129944 7816 consensus_queue.cc:260] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae721ac7f280489b99a961d9c2f6b90a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33287 } } peers { permanent_uuid: "ccb549cfef4a44e0a3b71499f1108953" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34363 } } peers { permanent_uuid: "a328d62d8cf743f5a7976113523df55f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43937 } }
I20260807 11:32:47.138453 7816 ts_tablet_manager.cc:1428] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953: Time spent starting tablet: real 0.022s user 0.006s sys 0.013s
I20260807 11:32:47.138935 7463 heartbeater.cc:503] Master 127.24.103.126:38957 was elected leader, sending a full tablet report...
I20260807 11:32:47.139273 7814 raft_consensus.cc:348] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae721ac7f280489b99a961d9c2f6b90a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33287 } } peers { permanent_uuid: "ccb549cfef4a44e0a3b71499f1108953" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34363 } } peers { permanent_uuid: "a328d62d8cf743f5a7976113523df55f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43937 } }
I20260807 11:32:47.139719 7814 raft_consensus.cc:374] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:32:47.139804 7814 raft_consensus.cc:729] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae721ac7f280489b99a961d9c2f6b90a, State: Initialized, Role: FOLLOWER
I20260807 11:32:47.140443 7814 consensus_queue.cc:260] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae721ac7f280489b99a961d9c2f6b90a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33287 } } peers { permanent_uuid: "ccb549cfef4a44e0a3b71499f1108953" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34363 } } peers { permanent_uuid: "a328d62d8cf743f5a7976113523df55f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43937 } }
I20260807 11:32:47.146502 7628 heartbeater.cc:503] Master 127.24.103.126:38957 was elected leader, sending a full tablet report...
I20260807 11:32:47.146852 7815 raft_consensus.cc:348] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae721ac7f280489b99a961d9c2f6b90a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33287 } } peers { permanent_uuid: "ccb549cfef4a44e0a3b71499f1108953" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34363 } } peers { permanent_uuid: "a328d62d8cf743f5a7976113523df55f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43937 } }
I20260807 11:32:47.147413 7815 raft_consensus.cc:374] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:32:47.147523 7815 raft_consensus.cc:729] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a328d62d8cf743f5a7976113523df55f, State: Initialized, Role: FOLLOWER
I20260807 11:32:47.148782 7814 ts_tablet_manager.cc:1428] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260807 11:32:47.151602 7815 consensus_queue.cc:260] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae721ac7f280489b99a961d9c2f6b90a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33287 } } peers { permanent_uuid: "ccb549cfef4a44e0a3b71499f1108953" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34363 } } peers { permanent_uuid: "a328d62d8cf743f5a7976113523df55f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43937 } }
I20260807 11:32:47.153103 7783 heartbeater.cc:503] Master 127.24.103.126:38957 was elected leader, sending a full tablet report...
I20260807 11:32:47.153836 7815 ts_tablet_manager.cc:1428] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: Time spent starting tablet: real 0.027s user 0.013s sys 0.002s
W20260807 11:32:47.259877 7464 tablet.cc:2369] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:32:47.323635 7820 raft_consensus.cc:482] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:32:47.324020 7820 raft_consensus.cc:504] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae721ac7f280489b99a961d9c2f6b90a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33287 } } peers { permanent_uuid: "ccb549cfef4a44e0a3b71499f1108953" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34363 } } peers { permanent_uuid: "a328d62d8cf743f5a7976113523df55f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43937 } }
I20260807 11:32:47.325749 7820 leader_election.cc:288] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae721ac7f280489b99a961d9c2f6b90a (127.24.103.66:33287), a328d62d8cf743f5a7976113523df55f (127.24.103.67:43937)
W20260807 11:32:47.329806 7785 tablet.cc:2369] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:32:47.333540 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.326328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:52537 (local address 127.24.103.67:43937)
0807 11:32:47.326589 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:47.326598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:47.327314 (+ 716us) server_negotiation.cc:407] Connection header received
0807 11:32:47.327527 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:47.327531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:47.327591 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:47.327694 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:32:47.328369 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.329071 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.331695 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.332464 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.332765 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.332846 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:47.333135 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 11:32:47.333274 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260807 11:32:47.333582 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.326275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:60939 (local address 127.24.103.66:33287)
0807 11:32:47.326512 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:47.326521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:47.327269 (+ 748us) server_negotiation.cc:407] Connection header received
0807 11:32:47.327502 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:47.327506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:47.327579 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:47.327702 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:32:47.328362 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.329061 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.331692 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.332469 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.332659 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.332749 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:47.333191 (+ 442us) server_negotiation.cc:299] Negotiation successful
0807 11:32:47.333327 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:32:47.333705 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.326266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:43937 (local address 127.24.103.65:52537)
0807 11:32:47.327186 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.327201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:47.327371 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:47.327717 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:47.327721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:47.327747 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:32:47.328192 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:47.328203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.329223 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.329230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:47.331529 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:47.331547 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.332851 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.332856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:47.332974 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.332998 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 11:32:47.333075 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 11:32:47.333227 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":90,"thread_start_us":427,"threads_started":1}
I20260807 11:32:47.333811 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.326223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:33287 (local address 127.24.103.65:60939)
0807 11:32:47.327127 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.327152 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:47.327346 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:47.327718 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:47.327722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:47.327816 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:32:47.328206 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:47.328214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.329226 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.329232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:47.331504 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:47.331521 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.332806 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.332814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:47.332996 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.333019 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 11:32:47.333141 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 11:32:47.333276 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":148,"spinlock_wait_cycles":24832,"thread_start_us":621,"threads_started":1}
I20260807 11:32:47.334851 7732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ec726b11044e62932fc7a56cfdb998" candidate_uuid: "ccb549cfef4a44e0a3b71499f1108953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a328d62d8cf743f5a7976113523df55f" is_pre_election: true
I20260807 11:32:47.334851 7576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ec726b11044e62932fc7a56cfdb998" candidate_uuid: "ccb549cfef4a44e0a3b71499f1108953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae721ac7f280489b99a961d9c2f6b90a" is_pre_election: true
I20260807 11:32:47.335467 7732 raft_consensus.cc:2454] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb549cfef4a44e0a3b71499f1108953 in term 0.
I20260807 11:32:47.335467 7576 raft_consensus.cc:2454] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb549cfef4a44e0a3b71499f1108953 in term 0.
I20260807 11:32:47.336300 7338 leader_election.cc:302] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a328d62d8cf743f5a7976113523df55f, ae721ac7f280489b99a961d9c2f6b90a, ccb549cfef4a44e0a3b71499f1108953; no voters:
I20260807 11:32:47.336764 7820 raft_consensus.cc:2781] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:32:47.336915 7820 raft_consensus.cc:482] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:32:47.337026 7820 raft_consensus.cc:3037] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:32:47.339288 7820 raft_consensus.cc:504] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae721ac7f280489b99a961d9c2f6b90a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33287 } } peers { permanent_uuid: "ccb549cfef4a44e0a3b71499f1108953" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34363 } } peers { permanent_uuid: "a328d62d8cf743f5a7976113523df55f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43937 } }
I20260807 11:32:47.339917 7820 leader_election.cc:288] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [CANDIDATE]: Term 1 election: Requested vote from peers ae721ac7f280489b99a961d9c2f6b90a (127.24.103.66:33287), a328d62d8cf743f5a7976113523df55f (127.24.103.67:43937)
I20260807 11:32:47.340440 7732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ec726b11044e62932fc7a56cfdb998" candidate_uuid: "ccb549cfef4a44e0a3b71499f1108953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a328d62d8cf743f5a7976113523df55f"
I20260807 11:32:47.340440 7576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ec726b11044e62932fc7a56cfdb998" candidate_uuid: "ccb549cfef4a44e0a3b71499f1108953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae721ac7f280489b99a961d9c2f6b90a"
I20260807 11:32:47.340660 7576 raft_consensus.cc:3037] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:32:47.340660 7732 raft_consensus.cc:3037] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:32:47.343976 7576 raft_consensus.cc:2454] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb549cfef4a44e0a3b71499f1108953 in term 1.
I20260807 11:32:47.343976 7732 raft_consensus.cc:2454] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb549cfef4a44e0a3b71499f1108953 in term 1.
I20260807 11:32:47.344456 7338 leader_election.cc:302] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a328d62d8cf743f5a7976113523df55f, ccb549cfef4a44e0a3b71499f1108953; no voters:
I20260807 11:32:47.344777 7820 raft_consensus.cc:2781] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:32:47.345284 7820 raft_consensus.cc:686] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 1 LEADER]: Becoming Leader. State: Replica: ccb549cfef4a44e0a3b71499f1108953, State: Running, Role: LEADER
I20260807 11:32:47.345870 7820 consensus_queue.cc:237] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae721ac7f280489b99a961d9c2f6b90a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33287 } } peers { permanent_uuid: "ccb549cfef4a44e0a3b71499f1108953" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34363 } } peers { permanent_uuid: "a328d62d8cf743f5a7976113523df55f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43937 } }
I20260807 11:32:47.350876 7243 catalog_manager.cc:5495] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccb549cfef4a44e0a3b71499f1108953 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "ccb549cfef4a44e0a3b71499f1108953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae721ac7f280489b99a961d9c2f6b90a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb549cfef4a44e0a3b71499f1108953" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a328d62d8cf743f5a7976113523df55f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
W20260807 11:32:47.358713 7631 tablet.cc:2369] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:32:47.391904 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.386287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48902 (local address 127.24.103.65:34363)
0807 11:32:47.386498 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:47.386506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:47.386528 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:32:47.386798 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:47.386802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:47.386892 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:47.387075 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:32:47.387525 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.388235 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.389645 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.390050 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.390185 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.390702 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:32:47.391282 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:32:47.391439 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:47.391549 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 11:32:47.391631 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 11:32:47.400339 7732 raft_consensus.cc:1261] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f [term 1 FOLLOWER]: Refusing update from remote peer ccb549cfef4a44e0a3b71499f1108953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:32:47.400336 7576 raft_consensus.cc:1261] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a [term 1 FOLLOWER]: Refusing update from remote peer ccb549cfef4a44e0a3b71499f1108953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:32:47.401345 7820 consensus_queue.cc:1035] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a328d62d8cf743f5a7976113523df55f" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:32:47.401676 7834 consensus_queue.cc:1035] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae721ac7f280489b99a961d9c2f6b90a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:32:47.430882 7243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:32:47.431252 7243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:32:47.435405 7243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43df95c82e404daaa4be1af94c494a25: Sending DeleteTablet for 3 replicas of tablet f4ec726b11044e62932fc7a56cfdb998
I20260807 11:32:47.436755 7390 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ec726b11044e62932fc7a56cfdb998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:32:47 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:48900
I20260807 11:32:47.436750 7552 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ec726b11044e62932fc7a56cfdb998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:32:47 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:34034
I20260807 11:32:47.436775 7710 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ec726b11044e62932fc7a56cfdb998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:32:47 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:50962
I20260807 11:32:47.443632 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 7284
I20260807 11:32:47.450066 7852 tablet_replica.cc:330] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: stopping tablet replica
I20260807 11:32:47.450788 7852 raft_consensus.cc:2229] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:32:47.451527 7852 raft_consensus.cc:2258] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:32:47.452236 7854 tablet_replica.cc:330] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: stopping tablet replica
I20260807 11:32:47.452872 7854 raft_consensus.cc:2229] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:32:47.453596 7854 raft_consensus.cc:2258] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:32:47.454864 7852 ts_tablet_manager.cc:1905] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:32:47.456885 7853 tablet_replica.cc:330] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953: stopping tablet replica
I20260807 11:32:47.457265 7853 raft_consensus.cc:2229] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:32:47.457898 7853 raft_consensus.cc:2258] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:32:47.459369 7854 ts_tablet_manager.cc:1905] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:32:47.462337 7852 ts_tablet_manager.cc:1918] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:32:47.464572 7853 ts_tablet_manager.cc:1905] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:32:47.465278 7852 log.cc:1196] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/f4ec726b11044e62932fc7a56cfdb998
I20260807 11:32:47.465917 7852 ts_tablet_manager.cc:1939] T f4ec726b11044e62932fc7a56cfdb998 P a328d62d8cf743f5a7976113523df55f: Deleting consensus metadata
I20260807 11:32:47.468271 7227 catalog_manager.cc:4841] TS a328d62d8cf743f5a7976113523df55f (127.24.103.67:43937): tablet f4ec726b11044e62932fc7a56cfdb998 (table test-table [id=9006bc92d7cc4c8da48081195c01f4cf]) successfully deleted
I20260807 11:32:47.469343 7854 ts_tablet_manager.cc:1918] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:32:47.469616 7854 log.cc:1196] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/f4ec726b11044e62932fc7a56cfdb998
I20260807 11:32:47.470242 7854 ts_tablet_manager.cc:1939] T f4ec726b11044e62932fc7a56cfdb998 P ae721ac7f280489b99a961d9c2f6b90a: Deleting consensus metadata
I20260807 11:32:47.472267 7228 catalog_manager.cc:4841] TS ae721ac7f280489b99a961d9c2f6b90a (127.24.103.66:33287): tablet f4ec726b11044e62932fc7a56cfdb998 (table test-table [id=9006bc92d7cc4c8da48081195c01f4cf]) successfully deleted
I20260807 11:32:47.473578 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.444788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.24.103.65:34363)
0807 11:32:47.445026 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:47.445055 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:47.452513 (+ 7458us) server_negotiation.cc:407] Connection header received
0807 11:32:47.452740 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:47.452746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:47.452836 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:47.452965 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:47.456448 (+ 3483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.457700 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.459625 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.460296 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.460434 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.464201 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:47.464216 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:47.464233 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:47.464298 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:47.469642 (+ 5344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:47.470539 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:47.470548 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:47.470551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:47.470736 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:47.471257 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:47.471267 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:47.471269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:47.471724 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:47.472054 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:47.472911 (+ 857us) server_negotiation.cc:299] Negotiation successful
0807 11:32:47.473184 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:32:47.474448 7458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:47.574908 7853 ts_tablet_manager.cc:1918] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:32:47.575350 7853 log.cc:1196] T f4ec726b11044e62932fc7a56cfdb998 P ccb549cfef4a44e0a3b71499f1108953: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/f4ec726b11044e62932fc7a56cfdb998
I20260807 11:32:47.575598 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 7284
W20260807 11:32:47.598702 7224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:34363 (error 108)
I20260807 11:32:47.598917 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 7467
I20260807 11:32:47.601164 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.600316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34363 (local address 127.0.0.1:48920)
0807 11:32:47.600856 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.600977 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":281,"threads_started":1}
W20260807 11:32:47.601432 7224 catalog_manager.cc:4584] TS ccb549cfef4a44e0a3b71499f1108953 (127.24.103.65:34363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4ec726b11044e62932fc7a56cfdb998: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34363: connect: Connection refused (error 111)
I20260807 11:32:47.613991 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.599843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.24.103.66:33287)
0807 11:32:47.600059 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:47.600064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:47.600168 (+ 104us) server_negotiation.cc:407] Connection header received
0807 11:32:47.600383 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:47.600387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:47.600462 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:47.600585 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:47.602094 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.602874 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.604352 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.604753 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.604897 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.607564 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:47.607578 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:47.607593 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:47.607632 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:47.610868 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:47.611761 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:47.611776 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:47.611779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:47.611928 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:47.612341 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:47.612347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:47.612350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:47.612723 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:47.613072 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:47.613383 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 11:32:47.613611 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:32:47.614732 7623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:47.655437 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.654942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34363 (local address 127.0.0.1:48934)
0807 11:32:47.655186 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.655290 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260807 11:32:47.656529 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.656144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34363 (local address 127.0.0.1:48944)
0807 11:32:47.656312 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.656393 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 11:32:47.705927 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 7467
I20260807 11:32:47.717221 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.716546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34363 (local address 127.0.0.1:48948)
0807 11:32:47.716864 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.716986 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260807 11:32:47.718540 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.718050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34363 (local address 127.0.0.1:48952)
0807 11:32:47.718254 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.718352 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 11:32:47.728739 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 7635
I20260807 11:32:47.743302 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.729628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.24.103.67:43937)
0807 11:32:47.729802 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:47.729807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:47.729847 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:32:47.729907 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:47.729911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:47.729975 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:47.730097 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:47.731610 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.732441 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.733835 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:47.734244 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:47.734385 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:47.737336 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:47.737351 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:47.737367 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:47.737409 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:47.740378 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:47.741093 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:47.741101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:47.741103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:47.741269 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:47.741639 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:47.741645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:47.741648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:47.742011 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:47.742324 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:47.742625 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 11:32:47.742842 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 11:32:47.744240 7778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:47.798998 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.798417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34363 (local address 127.0.0.1:48960)
0807 11:32:47.798682 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.798796 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260807 11:32:47.800180 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:47.799706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34363 (local address 127.0.0.1:48976)
0807 11:32:47.799921 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:47.800003 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260807 11:32:47.834874 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 7635
I20260807 11:32:47.857389 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 7204
I20260807 11:32:47.858731 7275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:47.974190 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 7204
I20260807 11:32:48.001276 24989 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 11:32:48.347254 24989 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 11:32:48.347508 24989 mini_ranger.cc:66] Stopping Ranger...
I20260807 11:32:48.693468 24989 mini_ranger.cc:68] Stopped Ranger
2026-08-07 11:32:48.693 UTC [6121] LOG: received smart shutdown request
2026-08-07 11:32:48.697 UTC [6121] LOG: background worker "logical replication launcher" (PID 6132) exited with exit code 1
2026-08-07 11:32:48.697 UTC [6127] LOG: shutting down
2026-08-07 11:32:48.893 UTC [6121] LOG: database system is shut down
2026-08-07T11:32:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57141 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:32:49 dist-test-slave-wl2h krb5kdc[7871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:32:49 dist-test-slave-wl2h krb5kdc[7871](info): set up 2 sockets
Aug 07 11:32:49 dist-test-slave-wl2h krb5kdc[7871](info): commencing operation
krb5kdc: starting...
W20260807 11:32:50.273784 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:32:50 dist-test-slave-wl2h krb5kdc[7871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:32:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:32:50.582975 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42067
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42067,127.24.103.125:34559,127.24.103.124:38247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:32:50.938489 7887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:50.938819 7887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:50.938987 7887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:50.946631 7887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:32:50.946758 7887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:50.946841 7887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:50.946938 7887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:32:50.947053 7887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:32:50.957551 7887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42067,127.24.103.125:34559,127.24.103.124:38247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:50.959101 7887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:50.960492 7887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:50.973151 7892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:50.973255 7893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:50.973197 7895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:50.974284 7887 server_base.cc:1034] running on GCE node
I20260807 11:32:50.975286 7887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:50.977079 7887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:50.978338 7887 hybrid_clock.cc:648] HybridClock initialized: now 1786102370978290 us; error 43 us; skew 500 ppm
I20260807 11:32:50.978793 7887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:50 dist-test-slave-wl2h krb5kdc[7871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102370, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:50.990314 7887 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:32:50.992383 7887 webserver.cc:460] Webserver started at http://127.24.103.126:45859/ using document root <none> and password file <none>
I20260807 11:32:50.993259 7887 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:50.993381 7887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:32:50.993703 7887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:32:50.997285 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "dcab552290df4f2cb2a9c12c22825c2e"
format_stamp: "Formatted at 2026-08-07 11:32:50 on dist-test-slave-wl2h"
server_key: "7a64c2a160223cc97902e7ab461669c8"
server_key_iv: "0d73ede1b713b168e7c4c468abffe397"
server_key_version: "encryptionkey@0"
I20260807 11:32:50.998149 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "dcab552290df4f2cb2a9c12c22825c2e"
format_stamp: "Formatted at 2026-08-07 11:32:50 on dist-test-slave-wl2h"
server_key: "7a64c2a160223cc97902e7ab461669c8"
server_key_iv: "0d73ede1b713b168e7c4c468abffe397"
server_key_version: "encryptionkey@0"
I20260807 11:32:51.004474 7887 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260807 11:32:51.008579 7903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:51.010311 7887 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 11:32:51.010617 7887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "dcab552290df4f2cb2a9c12c22825c2e"
format_stamp: "Formatted at 2026-08-07 11:32:50 on dist-test-slave-wl2h"
server_key: "7a64c2a160223cc97902e7ab461669c8"
server_key_iv: "0d73ede1b713b168e7c4c468abffe397"
server_key_version: "encryptionkey@0"
I20260807 11:32:51.010845 7887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:51.030375 7887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:51.035892 7887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:51.036267 7887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:51.059165 7887 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:42067
I20260807 11:32:51.059381 7963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:42067 every 8 connection(s)
I20260807 11:32:51.060956 7887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:32:51.064613 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 7887
I20260807 11:32:51.064881 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:32:51.065116 7964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:32:51.065757 24989 external_mini_cluster.cc:1428] Setting key 504ee88b4a0816e35328cd816c3c43e2
I20260807 11:32:51.071993 7964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:32:51 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102370, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.125@KRBTEST.COM" created.
I20260807 11:32:51.098695 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.075481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.24.103.126:42067)
0807 11:32:51.077495 (+ 2014us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:51.077501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:51.077544 (+ 43us) server_negotiation.cc:407] Connection header received
0807 11:32:51.079682 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:51.079693 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:51.080089 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:51.080312 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:51.081633 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.083684 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.085113 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.086176 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.086392 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:51.090196 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:51.090208 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:51.090227 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:51.090272 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:51.093917 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:51.094675 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:51.094682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:51.094684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:51.094850 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:51.095328 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:51.095334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:51.095338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:51.095880 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:51.096106 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:51.096686 (+ 580us) server_negotiation.cc:299] Negotiation successful
0807 11:32:51.097146 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":1808,"spinlock_wait_cycles":29952,"thread_start_us":1556,"threads_started":1}
I20260807 11:32:51.098717 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.075481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42067 (local address 127.0.0.1:53350)
0807 11:32:51.076492 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.076551 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:51.079272 (+ 2721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:51.080424 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:51.080429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:51.081029 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:51.081413 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:51.081432 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.083862 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.083866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:51.084954 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:51.084966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.086413 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.086418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:51.086535 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:51.087329 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:51.087353 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:51.089930 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:51.094071 (+ 4141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:51.094084 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:51.094096 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:51.094523 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:51.094976 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:51.094981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:51.094985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:51.095210 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:51.096072 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:51.096079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:51.096318 (+ 239us) client_negotiation.cc:769] Sending connection context
0807 11:32:51.096579 (+ 261us) client_negotiation.cc:240] Negotiation successful
0807 11:32:51.097259 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":208,"thread_start_us":425,"threads_started":1}
I20260807 11:32:51.102034 7964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:32:51.103370 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.102885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47626)
0807 11:32:51.103111 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.103219 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260807 11:32:51.103652 7907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
I20260807 11:32:51.105297 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.104922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47632)
0807 11:32:51.105093 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.105161 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 11:32:51.105501 7964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:34559: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
I20260807 11:32:51.136651 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:34559
--webserver_interface=127.24.103.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42067,127.24.103.125:34559,127.24.103.124:38247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:32:51.154886 7964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } attempt: 1
I20260807 11:32:51.156399 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.155832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47638)
0807 11:32:51.156125 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.156253 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260807 11:32:51.157688 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.157276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47646)
0807 11:32:51.157458 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.157533 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 11:32:51.157876 7964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:34559: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
I20260807 11:32:51.226156 7964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } attempt: 2
I20260807 11:32:51.227595 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.226983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47650)
0807 11:32:51.227315 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.227436 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260807 11:32:51.228796 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.228363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47656)
0807 11:32:51.228559 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.228652 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 11:32:51.229031 7964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:34559: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
I20260807 11:32:51.320329 7964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } attempt: 3
I20260807 11:32:51.321525 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.321039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47660)
0807 11:32:51.321262 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.321388 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 11:32:51.322832 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.322408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47670)
0807 11:32:51.322585 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.322673 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 11:32:51.323077 7964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:34559: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
I20260807 11:32:51.466415 7964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } attempt: 4
I20260807 11:32:51.468045 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.467396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47684)
0807 11:32:51.467788 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.467883 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260807 11:32:51.469370 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.468878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47700)
0807 11:32:51.469132 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.469224 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 11:32:51.469578 7964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:34559: Network error: Client connection negotiation failed: client connection to 127.24.103.125:34559: connect: Connection refused (error 111)
W20260807 11:32:51.496883 7971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:51.497313 7971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:51.497484 7971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:51.505321 7971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:32:51.505506 7971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:51.505606 7971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:51.505693 7971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:32:51.505781 7971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:32:51.516530 7971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42067,127.24.103.125:34559,127.24.103.124:38247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:34559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.103.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:51.518069 7971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:51.519670 7971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:51.532958 7976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:51.533198 7977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:51.533684 7979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:51.534308 7971 server_base.cc:1034] running on GCE node
I20260807 11:32:51.535223 7971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:51.536751 7971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:51.537990 7971 hybrid_clock.cc:648] HybridClock initialized: now 1786102371537923 us; error 51 us; skew 500 ppm
I20260807 11:32:51.538460 7971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:51 dist-test-slave-wl2h krb5kdc[7871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102371, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:51.542111 7971 init.cc:377] Logged in from keytab as kudu/127.24.103.125@KRBTEST.COM (short username kudu)
I20260807 11:32:51.543853 7971 webserver.cc:460] Webserver started at http://127.24.103.125:44283/ using document root <none> and password file <none>
I20260807 11:32:51.544602 7971 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:51.544744 7971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:32:51.545081 7971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:32:51.548683 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/data/instance:
uuid: "506415d3687a46369e297df14378dd39"
format_stamp: "Formatted at 2026-08-07 11:32:51 on dist-test-slave-wl2h"
server_key: "bdb124c689991a79ed94dd493234d9eb"
server_key_iv: "4fa13797363d8e0067f2845272d2cc4f"
server_key_version: "encryptionkey@0"
I20260807 11:32:51.549742 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/wal/instance:
uuid: "506415d3687a46369e297df14378dd39"
format_stamp: "Formatted at 2026-08-07 11:32:51 on dist-test-slave-wl2h"
server_key: "bdb124c689991a79ed94dd493234d9eb"
server_key_iv: "4fa13797363d8e0067f2845272d2cc4f"
server_key_version: "encryptionkey@0"
I20260807 11:32:51.556212 7971 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260807 11:32:51.560868 7986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:51.562659 7971 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:32:51.562917 7971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/wal
uuid: "506415d3687a46369e297df14378dd39"
format_stamp: "Formatted at 2026-08-07 11:32:51 on dist-test-slave-wl2h"
server_key: "bdb124c689991a79ed94dd493234d9eb"
server_key_iv: "4fa13797363d8e0067f2845272d2cc4f"
server_key_version: "encryptionkey@0"
I20260807 11:32:51.563235 7971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:51.580237 7971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:51.585502 7971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:51.585865 7971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:51.607671 7971 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.125:34559
I20260807 11:32:51.607707 8045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.125:34559 every 8 connection(s)
I20260807 11:32:51.609159 7971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
I20260807 11:32:51.613161 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:32:51.618963 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 7971
I20260807 11:32:51.619269 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-1/wal/instance
I20260807 11:32:51.619727 24989 external_mini_cluster.cc:1428] Setting key 979b0eeca3b33053c7bef763181ef3c1
I20260807 11:32:51.620378 8046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:32:51 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102371, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:51.645987 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.623128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.24.103.126:42067)
0807 11:32:51.623751 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:51.623760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:51.624985 (+ 1225us) server_negotiation.cc:407] Connection header received
0807 11:32:51.626834 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:51.626840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:51.626945 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:51.627092 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:51.628924 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.629779 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.632022 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.632653 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.632986 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:51.637953 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:51.637964 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:51.637972 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:51.638048 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:51.642149 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:51.643129 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:51.643137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:51.643143 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:51.643304 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:51.643771 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:51.643779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:51.643784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:51.644223 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:51.644380 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:51.645183 (+ 803us) server_negotiation.cc:299] Negotiation successful
0807 11:32:51.645465 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":278,"threads_started":1}
I20260807 11:32:51.646895 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.623739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42067 (local address 127.0.0.1:53362)
0807 11:32:51.624598 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.624661 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:51.626507 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:51.627346 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:51.627355 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:51.628017 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:51.628690 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:51.628709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.629972 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.629977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:51.631670 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:51.631691 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.633178 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.633186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:51.633469 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:51.634456 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:51.634485 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:51.637587 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:51.642360 (+ 4773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:51.642369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:51.642384 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:51.642943 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:51.643425 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:51.643429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:51.643431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:51.643637 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:51.644363 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:51.644370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:51.644755 (+ 385us) client_negotiation.cc:769] Sending connection context
0807 11:32:51.645015 (+ 260us) client_negotiation.cc:240] Negotiation successful
0807 11:32:51.645532 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":367,"threads_started":1}
I20260807 11:32:51.649129 8046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.103.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.124@KRBTEST.COM" created.
Aug 07 11:32:51 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102371, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260807 11:32:51.668892 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.650199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47706)
0807 11:32:51.650379 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.650402 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:51.650531 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:51.651440 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:51.651444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:51.651917 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:51.652075 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:51.652083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.653831 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.653835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:51.654830 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:51.654842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.656275 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.656283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:51.656530 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:51.657294 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:51.657323 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:51.660485 (+ 3162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:51.665311 (+ 4826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:51.665319 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:51.665326 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:51.665917 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:51.666417 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:51.666425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:51.666431 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:51.666648 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:51.667654 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:51.667659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:51.667859 (+ 200us) client_negotiation.cc:769] Sending connection context
0807 11:32:51.668020 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 11:32:51.668439 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 11:32:51.668979 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.650323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.24.103.125:34559)
0807 11:32:51.650964 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:51.650970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:51.650994 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:32:51.651100 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:51.651107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:51.651293 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:51.651453 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:51.652307 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.653652 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.655140 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.656063 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.656198 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:51.660811 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:51.660829 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:51.660845 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:51.660884 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:51.665111 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:51.666090 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:51.666098 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:51.666101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:51.666279 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:51.666848 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:51.666852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:51.666854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:51.667489 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:51.667756 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:51.668200 (+ 444us) server_negotiation.cc:299] Negotiation successful
0807 11:32:51.668578 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":492,"thread_start_us":335,"threads_started":1}
I20260807 11:32:51.671773 8046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:32:51.673281 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.672605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41238)
0807 11:32:51.672849 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.673010 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260807 11:32:51.673934 7988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
I20260807 11:32:51.676193 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.675606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41248)
0807 11:32:51.675807 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.675941 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 11:32:51.676513 8046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:38247: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
I20260807 11:32:51.695950 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:38247
--webserver_interface=127.24.103.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42067,127.24.103.125:34559,127.24.103.124:38247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:32:51.726042 8046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } attempt: 1
I20260807 11:32:51.727449 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.726918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41252)
0807 11:32:51.727170 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.727286 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 11:32:51.728561 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.728163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41264)
0807 11:32:51.728354 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.728448 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 11:32:51.728796 8046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:38247: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
I20260807 11:32:51.768909 7964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } attempt: 5
Aug 07 11:32:51 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102370, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260807 11:32:51.785804 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.769915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.24.103.125:34559)
0807 11:32:51.770112 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:51.770118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:51.770140 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:32:51.770429 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:51.770433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:51.770508 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:51.770597 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:51.771805 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.772545 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.774525 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.775315 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.775430 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:51.779722 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:51.779728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:51.779732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:51.779773 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:51.782815 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:51.783615 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:51.783622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:51.783631 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:51.783756 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:51.784191 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:51.784199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:51.784202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:51.784617 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:51.784784 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:51.785217 (+ 433us) server_negotiation.cc:299] Negotiation successful
0807 11:32:51.785448 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:32:51.785821 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.769737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47722)
0807 11:32:51.769996 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.770019 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:51.770169 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:51.770690 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:51.770695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:51.771312 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:51.771547 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:51.771560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.772720 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.772727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:51.774334 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:51.774350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:51.775498 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:51.775505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:51.775731 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:51.776446 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:51.776471 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:51.779434 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:51.783041 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:51.783050 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:51.783054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:51.783469 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:51.783878 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:51.783885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:51.783888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:51.784057 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:51.784756 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:51.784766 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:51.784928 (+ 162us) client_negotiation.cc:769] Sending connection context
0807 11:32:51.785076 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 11:32:51.785441 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260807 11:32:51.786998 7964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:32:51.788371 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.787874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41270)
0807 11:32:51.788056 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.788199 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 11:32:51.789386 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.789037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41278)
0807 11:32:51.789204 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.789278 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 11:32:51.789620 7964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:38247: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
I20260807 11:32:51.797088 8046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } attempt: 2
I20260807 11:32:51.798415 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.797867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41286)
0807 11:32:51.798102 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.798219 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 11:32:51.799666 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.799258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41300)
0807 11:32:51.799416 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.799502 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 11:32:51.799976 8046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:38247: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
I20260807 11:32:51.840857 7964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } attempt: 1
I20260807 11:32:51.842257 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.841590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41306)
0807 11:32:51.841858 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.842012 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 11:32:51.843580 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.843171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41308)
0807 11:32:51.843348 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.843438 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 11:32:51.843823 7964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:38247: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
I20260807 11:32:51.891253 8046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } attempt: 3
I20260807 11:32:51.892573 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.892064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41310)
0807 11:32:51.892303 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.892409 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 11:32:51.893712 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.893282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41318)
0807 11:32:51.893515 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.893592 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260807 11:32:51.894014 8046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:38247: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
I20260807 11:32:51.912107 7964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } attempt: 2
I20260807 11:32:51.913628 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.912948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41320)
0807 11:32:51.913252 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.913381 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 11:32:51.914851 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:51.914401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41330)
0807 11:32:51.914598 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:51.914708 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260807 11:32:51.915303 7964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:38247: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
I20260807 11:32:52.021764 7964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } attempt: 3
I20260807 11:32:52.023200 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.022507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41336)
0807 11:32:52.022786 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:52.022915 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260807 11:32:52.024564 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.024072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41340)
0807 11:32:52.024323 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:52.024417 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260807 11:32:52.024832 7964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:38247: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
I20260807 11:32:52.037256 8046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } attempt: 4
I20260807 11:32:52.038535 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.037996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41352)
0807 11:32:52.038264 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:52.038365 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 11:32:52.039768 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.039392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41366)
0807 11:32:52.039549 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:52.039628 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 11:32:52.040038 8046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:38247: Network error: Client connection negotiation failed: client connection to 127.24.103.124:38247: connect: Connection refused (error 111)
W20260807 11:32:52.061069 8057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:52.061442 8057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:52.061609 8057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:52.069378 8057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:32:52.069533 8057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:52.069634 8057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:52.069733 8057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:32:52.069816 8057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:32:52.080420 8057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42067,127.24.103.125:34559,127.24.103.124:38247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:38247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.103.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:52.081895 8057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:52.083338 8057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:52.095647 8063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:52.095721 8064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:52.095685 8066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:52.097285 8057 server_base.cc:1034] running on GCE node
I20260807 11:32:52.098162 8057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:52.099750 8057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:52.100998 8057 hybrid_clock.cc:648] HybridClock initialized: now 1786102372100958 us; error 55 us; skew 500 ppm
I20260807 11:32:52.101384 8057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:52 dist-test-slave-wl2h krb5kdc[7871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102372, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:52.104799 8057 init.cc:377] Logged in from keytab as kudu/127.24.103.124@KRBTEST.COM (short username kudu)
I20260807 11:32:52.106393 8057 webserver.cc:460] Webserver started at http://127.24.103.124:43135/ using document root <none> and password file <none>
I20260807 11:32:52.107126 8057 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:52.107235 8057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:32:52.107491 8057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:32:52.110471 8057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/data/instance:
uuid: "20410d6a4920420caaefff23732441c6"
format_stamp: "Formatted at 2026-08-07 11:32:52 on dist-test-slave-wl2h"
server_key: "816c2b07c3e6f7c7711d72b21098592a"
server_key_iv: "452ddd59448a616eccbb7036bf2b0045"
server_key_version: "encryptionkey@0"
I20260807 11:32:52.111334 8057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/wal/instance:
uuid: "20410d6a4920420caaefff23732441c6"
format_stamp: "Formatted at 2026-08-07 11:32:52 on dist-test-slave-wl2h"
server_key: "816c2b07c3e6f7c7711d72b21098592a"
server_key_iv: "452ddd59448a616eccbb7036bf2b0045"
server_key_version: "encryptionkey@0"
I20260807 11:32:52.116815 8057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:32:52.120391 8073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:52.121791 8057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:32:52.121996 8057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/wal
uuid: "20410d6a4920420caaefff23732441c6"
format_stamp: "Formatted at 2026-08-07 11:32:52 on dist-test-slave-wl2h"
server_key: "816c2b07c3e6f7c7711d72b21098592a"
server_key_iv: "452ddd59448a616eccbb7036bf2b0045"
server_key_version: "encryptionkey@0"
I20260807 11:32:52.122216 8057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:52.137171 8057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:52.141824 8057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:52.142146 8057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:52.162827 8057 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.124:38247
I20260807 11:32:52.162880 8130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.124:38247 every 8 connection(s)
I20260807 11:32:52.164466 8057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
I20260807 11:32:52.167348 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 8057
I20260807 11:32:52.167585 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786102153172159-24989-0/minicluster-data/master-2/wal/instance
I20260807 11:32:52.168244 24989 external_mini_cluster.cc:1428] Setting key ab46012de9ccdded5b3758983ab27300
I20260807 11:32:52.168658 8132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:32:52.178248 8132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:32:52 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
Aug 07 11:32:52 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102372, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:52.195935 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.170197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.24.103.124:38247)
0807 11:32:52.171106 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:52.171124 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:52.171491 (+ 367us) server_negotiation.cc:407] Connection header received
0807 11:32:52.173933 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:52.173957 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:52.174402 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:52.174848 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:52.176404 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.177678 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.179202 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.180247 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.180444 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.186187 (+ 5743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:52.186203 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:52.186225 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:52.186290 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:52.190875 (+ 4585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.192101 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.192109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.192117 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.192370 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.192811 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.192816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.192820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.193443 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:52.193663 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:52.194290 (+ 627us) server_negotiation.cc:299] Negotiation successful
0807 11:32:52.194695 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":403,"threads_started":1}
I20260807 11:32:52.199751 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.181436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42067 (local address 127.0.0.1:53374)
0807 11:32:52.182499 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:52.182553 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:52.182733 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:52.183249 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:52.183255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:52.183988 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:52.184297 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:52.184307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.185973 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.185979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:52.187911 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:52.187933 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.189028 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.189035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:52.189235 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.189887 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:52.189918 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:52.192945 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:52.196627 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:52.196638 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:52.196644 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:52.197225 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:52.197715 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:52.197718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:52.197720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:52.197862 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:52.198521 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:52.198529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:52.198775 (+ 246us) client_negotiation.cc:769] Sending connection context
0807 11:32:52.198957 (+ 182us) client_negotiation.cc:240] Negotiation successful
0807 11:32:52.199337 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":643,"threads_started":1}
I20260807 11:32:52.199901 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.181487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.24.103.126:42067)
0807 11:32:52.182082 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:52.182089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:52.182683 (+ 594us) server_negotiation.cc:407] Connection header received
0807 11:32:52.182884 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:52.182891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:52.182989 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:52.183146 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:52.184481 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.185769 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.188145 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.188795 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.188904 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.193183 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:52.193190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:52.193193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:52.193230 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:52.196492 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.197368 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.197374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.197377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.197581 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.197979 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.197985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.197988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.198370 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:52.198524 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:52.199187 (+ 663us) server_negotiation.cc:299] Negotiation successful
0807 11:32:52.199433 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":290,"threads_started":1}
I20260807 11:32:52.201555 8132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:32:52.202128 7964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } attempt: 4
Aug 07 11:32:52 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
Aug 07 11:32:52 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102372, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
Aug 07 11:32:52 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102370, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260807 11:32:52.218984 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.203003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.24.103.124:38247)
0807 11:32:52.203210 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:52.203218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:52.203235 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:32:52.203284 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:52.203289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:52.203372 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:52.203468 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:52.204271 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.205128 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.207257 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.207850 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.207978 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.212990 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:52.213001 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:52.213006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:52.213047 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:52.216222 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.216882 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.216887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.216891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.217030 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.217481 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.217486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.217489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.217828 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:52.217982 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:52.218396 (+ 414us) server_negotiation.cc:299] Negotiation successful
0807 11:32:52.218630 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 11:32:52.219053 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.202658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41380)
0807 11:32:52.202847 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:52.202864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:52.202954 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:52.203516 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:52.203522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:52.203955 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:52.204128 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:52.204136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.205286 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.205295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:52.207094 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:52.207112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.208026 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.208033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:52.208311 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.208975 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:52.209000 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:52.212799 (+ 3799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:52.216373 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:52.216379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:52.216382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:52.216752 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:52.217145 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:52.217149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:52.217153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:52.217360 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:52.217950 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:52.217954 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:52.218143 (+ 189us) client_negotiation.cc:769] Sending connection context
0807 11:32:52.218295 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 11:32:52.218624 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":29}
I20260807 11:32:52.219650 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.202353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.24.103.125:34559)
0807 11:32:52.202552 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:52.202558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:52.202724 (+ 166us) server_negotiation.cc:407] Connection header received
0807 11:32:52.202875 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:52.202883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:52.202958 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:52.203129 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:52.204069 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.205125 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.207212 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.207947 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.208056 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.212454 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:52.212467 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:52.212472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:52.212514 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:52.216309 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.217278 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.217283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.217286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.217448 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.217978 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.217983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.217986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.218309 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:52.218424 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:52.218987 (+ 563us) server_negotiation.cc:299] Negotiation successful
0807 11:32:52.219254 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:32:52.219779 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.199539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.24.103.126:42067)
0807 11:32:52.200389 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:52.200397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:52.200420 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:32:52.200494 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:52.200502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:52.200665 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:52.200813 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:52.203378 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.204494 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.206155 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.206613 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.206779 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.212061 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:52.212068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:52.212073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:52.212110 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:52.216130 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.217150 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.217156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.217159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.217345 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.217851 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.217857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.217859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.218252 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:52.218532 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:52.219189 (+ 657us) server_negotiation.cc:299] Negotiation successful
0807 11:32:52.219405 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":695,"thread_start_us":1567,"threads_started":1}
I20260807 11:32:52.220530 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.202371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:34559 (local address 127.0.0.1:47728)
0807 11:32:52.202624 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:52.202644 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:52.202751 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:52.203148 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:52.203153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:52.203691 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:52.203883 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:52.203894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.205402 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.205410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:52.206973 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:52.206992 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.208451 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.208460 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:52.208714 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.209294 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:52.209325 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:52.212118 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:52.216524 (+ 4406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:52.216530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:52.216534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:52.217127 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:52.217618 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:52.217622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:52.217626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:52.217847 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:52.218526 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:52.218530 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:52.218701 (+ 171us) client_negotiation.cc:769] Sending connection context
0807 11:32:52.219844 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0807 11:32:52.220147 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"spinlock_wait_cycles":31104}
I20260807 11:32:52.221921 8132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:32:52.225533 7964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e: Bootstrap starting.
I20260807 11:32:52.229955 7964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e: Neither blocks nor log segments found. Creating new log.
Aug 07 11:32:52 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102372, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260807 11:32:52.231472 7964 log.cc:824] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e: Log is configured to *not* fsync() on all Append() calls
Aug 07 11:32:52 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260807 11:32:52.236970 7964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e: No bootstrap required, opened a new log
I20260807 11:32:52.239233 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.222786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41396)
0807 11:32:52.223165 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:52.223182 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:52.223272 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:52.224844 (+ 1572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:52.224848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:52.225246 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:52.225385 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:52.225393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.226851 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.226860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:52.228051 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:52.228064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.228824 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.228830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:52.229039 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.229675 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:52.229692 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:52.231933 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:52.236189 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:52.236194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:52.236198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:52.236719 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:52.237176 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:52.237180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:52.237184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:52.237362 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:52.238119 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:52.238124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:52.238292 (+ 168us) client_negotiation.cc:769] Sending connection context
0807 11:32:52.238430 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 11:32:52.238779 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260807 11:32:52.240268 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.223156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.24.103.124:38247)
0807 11:32:52.224611 (+ 1455us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:52.224615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:52.224629 (+ 14us) server_negotiation.cc:407] Connection header received
0807 11:32:52.224664 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:52.224667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:52.224717 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:52.224826 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:52.225519 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.226707 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.228211 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.228662 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.228806 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.232163 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:52.232168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:52.232171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:52.232201 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:52.235236 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.236881 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.236887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.236890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.237048 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.237519 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.237526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.237531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.237925 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:52.238071 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:52.239705 (+ 1634us) server_negotiation.cc:299] Negotiation successful
0807 11:32:52.239930 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1339}
I20260807 11:32:52.241358 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.222614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.24.103.125:34559)
0807 11:32:52.222772 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:52.222779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:52.222796 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:32:52.222925 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:52.222930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:52.223052 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:52.223143 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:52.224541 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.225640 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.227550 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.228233 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.228332 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.234312 (+ 5980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:52.234319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:52.234324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:52.234398 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:52.237825 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.238737 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.238747 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.238751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.238921 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.239461 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.239469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.239473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.239780 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:52.240103 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:52.240678 (+ 575us) server_negotiation.cc:299] Negotiation successful
0807 11:32:52.240937 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 11:32:52.247084 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:32:52.247330 8132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6: Bootstrap starting.
I20260807 11:32:52.247483 7964 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } }
I20260807 11:32:52.247848 7964 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:32:52.247974 7964 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcab552290df4f2cb2a9c12c22825c2e, State: Initialized, Role: FOLLOWER
I20260807 11:32:52.248616 7964 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } }
I20260807 11:32:52.249568 8143 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } } }
I20260807 11:32:52.250054 8143 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:32:52.250802 7964 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:32:52.251888 8132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6: Neither blocks nor log segments found. Creating new log.
I20260807 11:32:52.254081 8132 log.cc:824] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6: Log is configured to *not* fsync() on all Append() calls
I20260807 11:32:52.258641 8132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6: No bootstrap required, opened a new log
I20260807 11:32:52.273159 8132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } }
I20260807 11:32:52.273931 8132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:32:52.274149 8132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20410d6a4920420caaefff23732441c6, State: Initialized, Role: FOLLOWER
I20260807 11:32:52.275185 8132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } }
I20260807 11:32:52.282354 8166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } } }
I20260807 11:32:52.284871 8166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:32:52.284742 8132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 11:32:52.291265 7910 tablet.cc:2369] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:32:52.297062 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.270266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.24.103.124:38247)
0807 11:32:52.270526 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:52.270534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:52.277221 (+ 6687us) server_negotiation.cc:407] Connection header received
0807 11:32:52.277456 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:52.277464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:52.277563 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:52.277690 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:52.280063 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.281610 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.283712 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.284418 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.284517 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.288542 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:52.288549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:52.288553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:52.288593 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:52.292843 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.294062 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.294071 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.294076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.294292 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.294903 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.294911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.294917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.295474 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:52.295735 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:52.296426 (+ 691us) server_negotiation.cc:299] Negotiation successful
0807 11:32:52.296678 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260807 11:32:52.301970 8167 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260807 11:32:52.305585 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.262698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.24.103.126:42067)
0807 11:32:52.262986 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:52.262996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:52.275510 (+ 12514us) server_negotiation.cc:407] Connection header received
0807 11:32:52.277505 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:52.277515 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:52.277692 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:52.277787 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:52.288644 (+ 10857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.289786 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.291854 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.292616 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.292725 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.296664 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:52.296671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:52.296676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:52.296715 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:52.301093 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.302100 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.302105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.302110 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.302306 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.302965 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.302973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.302979 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.303560 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:52.303892 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:52.304633 (+ 741us) server_negotiation.cc:299] Negotiation successful
0807 11:32:52.304978 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260807 11:32:52.305369 8167 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:32:52.294893 7887 master_runner.cc:186] Error getting master registration for 127.24.103.125:34559: OK, instance_id { permanent_uuid: "506415d3687a46369e297df14378dd39" instance_seqno: 1786102371601694 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:32:52.307164 7887 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 11:32:52.312630 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.263797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.24.103.125:34559)
0807 11:32:52.263975 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:52.263980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:52.276860 (+ 12880us) server_negotiation.cc:407] Connection header received
0807 11:32:52.279107 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:52.279127 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:52.279240 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:52.279340 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:52.281541 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.283083 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.285848 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.286803 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.286928 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.304951 (+ 18023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:52.304958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:52.304963 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:52.305009 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:52.308680 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.309718 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.309727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.309733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.309921 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.310503 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.310511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.310515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.310961 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:52.311259 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:52.312005 (+ 746us) server_negotiation.cc:299] Negotiation successful
0807 11:32:52.312285 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 11:32:52.316504 8181 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:32:52.316685 8181 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:32:52.318089 8057 master_runner.cc:186] Error getting master registration for 127.24.103.125:34559: OK, instance_id { permanent_uuid: "506415d3687a46369e297df14378dd39" instance_seqno: 1786102371601694 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:32:52.329725 8057 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 11:32:52.339291 8046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } attempt: 5
Aug 07 11:32:52 dist-test-slave-wl2h krb5kdc[7871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102371, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260807 11:32:52.360142 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.339927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:38247 (local address 127.0.0.1:41406)
0807 11:32:52.340186 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:52.340206 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:52.340328 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:52.340702 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:52.340707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:52.341267 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:52.341482 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:52.341495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.343054 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.343060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:52.344531 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:52.344546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.345822 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.345829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:52.346090 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.346821 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:52.346846 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:52.349688 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:52.354218 (+ 4530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:52.354224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:52.354228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:52.354731 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:52.355259 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:52.355265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:52.355269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:52.355468 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:52.358975 (+ 3507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:52.358980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:52.359196 (+ 216us) client_negotiation.cc:769] Sending connection context
0807 11:32:52.359382 (+ 186us) client_negotiation.cc:240] Negotiation successful
0807 11:32:52.359710 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260807 11:32:52.360215 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:52.340018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.24.103.124:38247)
0807 11:32:52.340194 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:52.340199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:52.340292 (+ 93us) server_negotiation.cc:407] Connection header received
0807 11:32:52.340451 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:52.340456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:52.340540 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:52.340651 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:52.341665 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.342858 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.344704 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:52.345455 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:52.345595 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:52.349954 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:52.349961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:52.349966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:52.350007 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:52.354005 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.354875 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.354881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.354885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.355128 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:52.355609 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:52.355614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:52.355618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:52.356006 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:52.358904 (+ 2898us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:52.359491 (+ 587us) server_negotiation.cc:299] Negotiation successful
0807 11:32:52.359769 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 11:32:52.366647 8046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39: Bootstrap starting.
I20260807 11:32:52.371202 8046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39: Neither blocks nor log segments found. Creating new log.
I20260807 11:32:52.372694 8046 log.cc:824] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39: Log is configured to *not* fsync() on all Append() calls
I20260807 11:32:52.377158 8046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39: No bootstrap required, opened a new log
I20260807 11:32:52.389040 8046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } }
I20260807 11:32:52.389642 8046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:32:52.389813 8046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506415d3687a46369e297df14378dd39, State: Initialized, Role: FOLLOWER
I20260807 11:32:52.390518 8046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } }
I20260807 11:32:52.391630 8184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } } }
I20260807 11:32:52.392225 8184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:32:52.393049 8046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 11:32:52.394462 8080 tablet.cc:2369] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:32:52.415175 7971 master_runner.cc:263] No leader master found from master 506415d3687a46369e297df14378dd39
I20260807 11:32:52.415356 7971 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260807 11:32:52.421892 8197 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:32:52.422178 8197 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:32:52.474743 8143 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:32:52.475119 8143 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } }
I20260807 11:32:52.476994 8143 leader_election.cc:288] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506415d3687a46369e297df14378dd39 (127.24.103.125:34559), 20410d6a4920420caaefff23732441c6 (127.24.103.124:38247)
I20260807 11:32:52.478289 8016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcab552290df4f2cb2a9c12c22825c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506415d3687a46369e297df14378dd39" is_pre_election: true
I20260807 11:32:52.480216 8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcab552290df4f2cb2a9c12c22825c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20410d6a4920420caaefff23732441c6" is_pre_election: true
I20260807 11:32:52.480851 8102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcab552290df4f2cb2a9c12c22825c2e in term 0.
I20260807 11:32:52.478933 8016 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcab552290df4f2cb2a9c12c22825c2e in term 0.
I20260807 11:32:52.482281 7907 leader_election.cc:302] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506415d3687a46369e297df14378dd39, dcab552290df4f2cb2a9c12c22825c2e; no voters:
I20260807 11:32:52.483349 8143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:32:52.483520 8143 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:32:52.483628 8143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:32:52.489076 8143 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } }
I20260807 11:32:52.493331 8016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcab552290df4f2cb2a9c12c22825c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506415d3687a46369e297df14378dd39"
I20260807 11:32:52.493616 8016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:32:52.497581 8016 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcab552290df4f2cb2a9c12c22825c2e in term 1.
I20260807 11:32:52.498260 7907 leader_election.cc:302] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506415d3687a46369e297df14378dd39, dcab552290df4f2cb2a9c12c22825c2e; no voters:
I20260807 11:32:52.502590 8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcab552290df4f2cb2a9c12c22825c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20410d6a4920420caaefff23732441c6"
I20260807 11:32:52.502853 8102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:32:52.506562 8102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcab552290df4f2cb2a9c12c22825c2e in term 1.
I20260807 11:32:52.507542 8143 leader_election.cc:288] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [CANDIDATE]: Term 1 election: Requested vote from peers 506415d3687a46369e297df14378dd39 (127.24.103.125:34559), 20410d6a4920420caaefff23732441c6 (127.24.103.124:38247)
I20260807 11:32:52.508014 8143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:32:52.508667 8143 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [term 1 LEADER]: Becoming Leader. State: Replica: dcab552290df4f2cb2a9c12c22825c2e, State: Running, Role: LEADER
I20260807 11:32:52.514091 8143 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } }
I20260807 11:32:52.524919 8199 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcab552290df4f2cb2a9c12c22825c2e. Latest consensus state: current_term: 1 leader_uuid: "dcab552290df4f2cb2a9c12c22825c2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } } }
I20260807 11:32:52.525651 8199 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [sys.catalog]: This master's current role is: LEADER
I20260807 11:32:52.540045 8201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:32:52.542318 8201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:32:52.552989 8102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [term 1 FOLLOWER]: Refusing update from remote peer dcab552290df4f2cb2a9c12c22825c2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:32:52.554380 8199 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:32:52.562101 8016 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [term 1 FOLLOWER]: Refusing update from remote peer dcab552290df4f2cb2a9c12c22825c2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:32:52.563530 8199 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:32:52.583973 8166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcab552290df4f2cb2a9c12c22825c2e. Latest consensus state: current_term: 1 leader_uuid: "dcab552290df4f2cb2a9c12c22825c2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } } }
I20260807 11:32:52.584324 8166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:32:52.585978 8184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcab552290df4f2cb2a9c12c22825c2e. Latest consensus state: current_term: 1 leader_uuid: "dcab552290df4f2cb2a9c12c22825c2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } } }
I20260807 11:32:52.586328 8184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 11:32:52.589310 7994 tablet.cc:2369] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:32:52.589839 8202 mvcc.cc:205] Tried to move back new op lower bound from 7315875317964947456 to 7315875317830922240. Current Snapshot: MvccSnapshot[applied={T|T < 7315875317964947456}]
I20260807 11:32:52.597504 8204 mvcc.cc:205] Tried to move back new op lower bound from 7315875317964947456 to 7315875317830922240. Current Snapshot: MvccSnapshot[applied={T|T < 7315875317964947456}]
I20260807 11:32:52.599581 8143 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcab552290df4f2cb2a9c12c22825c2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } } }
I20260807 11:32:52.600052 8143 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [sys.catalog]: This master's current role is: LEADER
I20260807 11:32:52.600414 8143 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcab552290df4f2cb2a9c12c22825c2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } } }
I20260807 11:32:52.600782 8143 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e [sys.catalog]: This master's current role is: LEADER
I20260807 11:32:52.601749 8201 catalog_manager.cc:1347] Generated new cluster ID: cf030fc55cd246de97cef3737e520c92
I20260807 11:32:52.601859 8201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:32:52.613780 8184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcab552290df4f2cb2a9c12c22825c2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } } }
I20260807 11:32:52.614082 8184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:32:52.615859 8166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcab552290df4f2cb2a9c12c22825c2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcab552290df4f2cb2a9c12c22825c2e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42067 } } peers { permanent_uuid: "506415d3687a46369e297df14378dd39" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 34559 } } peers { permanent_uuid: "20410d6a4920420caaefff23732441c6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 38247 } } }
I20260807 11:32:52.616159 8166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:32:52.619899 8201 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:32:52.621728 8201 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:32:52.650295 8201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcab552290df4f2cb2a9c12c22825c2e: Generated new TSK 0
I20260807 11:32:52.651470 8201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:32:53.318838 8181 catalog_manager.cc:1259] Loaded cluster ID: cf030fc55cd246de97cef3737e520c92
I20260807 11:32:53.318956 8181 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6: loading cluster ID for follower catalog manager: success
I20260807 11:32:53.321542 8181 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6: acquiring CA information for follower catalog manager: success
I20260807 11:32:53.323448 8181 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20410d6a4920420caaefff23732441c6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:32:53.424369 8197 catalog_manager.cc:1259] Loaded cluster ID: cf030fc55cd246de97cef3737e520c92
I20260807 11:32:53.424562 8197 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39: loading cluster ID for follower catalog manager: success
I20260807 11:32:53.427610 8197 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39: acquiring CA information for follower catalog manager: success
I20260807 11:32:53.429492 8197 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 506415d3687a46369e297df14378dd39: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:32:53.803318 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 7887
I20260807 11:32:53.804661 7959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:53.923408 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 7887
W20260807 11:32:53.962833 8074 connection.cc:535] server connection from 127.0.0.1:41380 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 11:32:53.963547 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 7971
I20260807 11:32:53.965003 8041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:54.074527 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 7971
I20260807 11:32:54.099198 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 8057
I20260807 11:32:54.100678 8125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:54.213343 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 8057
2026-08-07T11:32:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5180 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:32:54 dist-test-slave-wl2h krb5kdc[8221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:32:54 dist-test-slave-wl2h krb5kdc[8221](info): set up 2 sockets
Aug 07 11:32:54 dist-test-slave-wl2h krb5kdc[8221](info): commencing operation
krb5kdc: starting...
W20260807 11:32:55.470510 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:32:55 dist-test-slave-wl2h krb5kdc[8221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:32:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:32:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:32:55.780319 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42493
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:32:56.136482 8236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:56.136795 8236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:56.136957 8236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:56.144550 8236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:32:56.144675 8236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:56.144766 8236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:56.144882 8236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:32:56.144965 8236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:32:56.155596 8236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:56.156968 8236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:56.158330 8236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:56.170692 8241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:56.170730 8244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:56.170653 8242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:56.171926 8236 server_base.cc:1034] running on GCE node
I20260807 11:32:56.172776 8236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:56.174289 8236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:56.175549 8236 hybrid_clock.cc:648] HybridClock initialized: now 1786102376175485 us; error 39 us; skew 500 ppm
I20260807 11:32:56.175940 8236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:56 dist-test-slave-wl2h krb5kdc[8221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102376, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:56.184960 8236 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:32:56.186758 8236 webserver.cc:460] Webserver started at http://127.24.103.126:40185/ using document root <none> and password file <none>
I20260807 11:32:56.187564 8236 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:56.187728 8236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:32:56.188057 8236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:32:56.191327 8236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "b6532fd6c99f47e3a6011bbc9f2ffad8"
format_stamp: "Formatted at 2026-08-07 11:32:56 on dist-test-slave-wl2h"
server_key: "2feec2116e1bc4c5028fc71462e868dd"
server_key_iv: "43ee757c2a632a5e5405d34f0532911f"
server_key_version: "encryptionkey@0"
I20260807 11:32:56.192171 8236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "b6532fd6c99f47e3a6011bbc9f2ffad8"
format_stamp: "Formatted at 2026-08-07 11:32:56 on dist-test-slave-wl2h"
server_key: "2feec2116e1bc4c5028fc71462e868dd"
server_key_iv: "43ee757c2a632a5e5405d34f0532911f"
server_key_version: "encryptionkey@0"
I20260807 11:32:56.197798 8236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 11:32:56.201609 8251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:56.202957 8236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:32:56.203219 8236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "b6532fd6c99f47e3a6011bbc9f2ffad8"
format_stamp: "Formatted at 2026-08-07 11:32:56 on dist-test-slave-wl2h"
server_key: "2feec2116e1bc4c5028fc71462e868dd"
server_key_iv: "43ee757c2a632a5e5405d34f0532911f"
server_key_version: "encryptionkey@0"
I20260807 11:32:56.203454 8236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:56.222052 8236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:56.226928 8236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:56.227327 8236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:56.249503 8236 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:42493
I20260807 11:32:56.249540 8311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:42493 every 8 connection(s)
I20260807 11:32:56.250989 8236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:32:56.254920 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:32:56.261036 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 8236
I20260807 11:32:56.261350 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:32:56.262132 24989 external_mini_cluster.cc:1428] Setting key 05c4e83b4431eeef28a5ed3e48c242f7
I20260807 11:32:56.266412 8312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8: Bootstrap starting.
I20260807 11:32:56.270402 8312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8: Neither blocks nor log segments found. Creating new log.
I20260807 11:32:56.271837 8312 log.cc:824] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8: Log is configured to *not* fsync() on all Append() calls
I20260807 11:32:56.274953 8312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8: No bootstrap required, opened a new log
Aug 07 11:32:56 dist-test-slave-wl2h krb5kdc[8221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:56.284216 8312 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6532fd6c99f47e3a6011bbc9f2ffad8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42493 } }
I20260807 11:32:56.284646 8312 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:32:56.284770 8312 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6532fd6c99f47e3a6011bbc9f2ffad8, State: Initialized, Role: FOLLOWER
I20260807 11:32:56.285449 8312 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6532fd6c99f47e3a6011bbc9f2ffad8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42493 } }
I20260807 11:32:56.285737 8312 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:32:56.285903 8312 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:32:56.286113 8312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:32:56.288671 8312 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6532fd6c99f47e3a6011bbc9f2ffad8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42493 } }
I20260807 11:32:56.288836 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:56.264389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.24.103.126:42493)
0807 11:32:56.265217 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:56.265233 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:56.265284 (+ 51us) server_negotiation.cc:407] Connection header received
0807 11:32:56.267886 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:56.267915 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:56.268468 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:56.268949 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:56.270460 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:56.272604 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:56.274101 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:56.274793 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:56.275010 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:56.279602 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:56.279611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:56.279626 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:56.279716 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:56.283978 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:56.284838 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:56.284847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:56.284856 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:56.285077 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:56.285537 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:56.285542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:56.285546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:56.286331 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:56.286655 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:56.287240 (+ 585us) server_negotiation.cc:299] Negotiation successful
0807 11:32:56.287774 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":359,"threads_started":1}
I20260807 11:32:56.289270 8312 leader_election.cc:302] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6532fd6c99f47e3a6011bbc9f2ffad8; no voters:
I20260807 11:32:56.289906 8312 leader_election.cc:288] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:32:56.290063 8317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:32:56.291096 8317 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [term 1 LEADER]: Becoming Leader. State: Replica: b6532fd6c99f47e3a6011bbc9f2ffad8, State: Running, Role: LEADER
I20260807 11:32:56.291718 8317 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6532fd6c99f47e3a6011bbc9f2ffad8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42493 } }
I20260807 11:32:56.292444 8312 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:32:56.293259 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:32:56.294670 8319 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6532fd6c99f47e3a6011bbc9f2ffad8. Latest consensus state: current_term: 1 leader_uuid: "b6532fd6c99f47e3a6011bbc9f2ffad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6532fd6c99f47e3a6011bbc9f2ffad8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42493 } } }
I20260807 11:32:56.294747 8318 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6532fd6c99f47e3a6011bbc9f2ffad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6532fd6c99f47e3a6011bbc9f2ffad8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42493 } } }
I20260807 11:32:56.295152 8319 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [sys.catalog]: This master's current role is: LEADER
I20260807 11:32:56.295176 8318 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8 [sys.catalog]: This master's current role is: LEADER
I20260807 11:32:56.298138 8323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:32:56.303673 8323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:32:56.313892 8323 catalog_manager.cc:1347] Generated new cluster ID: 27162243e4cb4af9b139bc55fe00fcbf
I20260807 11:32:56.314004 8323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:32:56.323314 8323 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:32:56.324687 8323 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:32:56.335397 8323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6532fd6c99f47e3a6011bbc9f2ffad8: Generated new TSK 0
I20260807 11:32:56.336371 8323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:32:56.356765 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 8236
I20260807 11:32:56.358110 8307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:32:56.455214 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 8236
2026-08-07T11:32:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2239 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:32:56 dist-test-slave-wl2h krb5kdc[8342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:32:56 dist-test-slave-wl2h krb5kdc[8342](info): set up 2 sockets
Aug 07 11:32:56 dist-test-slave-wl2h krb5kdc[8342](info): commencing operation
krb5kdc: starting...
W20260807 11:32:57.670907 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.003s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:32:57 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:32:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:32:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:32:57.984655 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42305
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:32:58.331794 8355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:58.332135 8355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:58.332312 8355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:58.339686 8355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:32:58.339798 8355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:58.339900 8355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:58.340009 8355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:32:58.340101 8355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:32:58.350183 8355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:58.351694 8355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:58.353039 8355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:58.365168 8367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:58.365218 8363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:58.365406 8362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:58.366261 8355 server_base.cc:1034] running on GCE node
I20260807 11:32:58.366971 8355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:58.368469 8355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:58.369851 8355 hybrid_clock.cc:648] HybridClock initialized: now 1786102378369778 us; error 51 us; skew 500 ppm
I20260807 11:32:58.370179 8355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:58 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102378, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:58.373448 8355 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:32:58.375180 8355 webserver.cc:460] Webserver started at http://127.24.103.126:34591/ using document root <none> and password file <none>
I20260807 11:32:58.375866 8355 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:58.375973 8355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:32:58.376315 8355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:32:58.379413 8355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "a5f9fa9bf2fc43108bf298e006972b3e"
format_stamp: "Formatted at 2026-08-07 11:32:58 on dist-test-slave-wl2h"
server_key: "9bccf25cbbc856a944056346a5ce1bae"
server_key_iv: "6fcdcbcca7679173256351b218f13094"
server_key_version: "encryptionkey@0"
I20260807 11:32:58.380139 8355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "a5f9fa9bf2fc43108bf298e006972b3e"
format_stamp: "Formatted at 2026-08-07 11:32:58 on dist-test-slave-wl2h"
server_key: "9bccf25cbbc856a944056346a5ce1bae"
server_key_iv: "6fcdcbcca7679173256351b218f13094"
server_key_version: "encryptionkey@0"
I20260807 11:32:58.385723 8355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 11:32:58.389667 8375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:58.391374 8355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:32:58.391587 8355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "a5f9fa9bf2fc43108bf298e006972b3e"
format_stamp: "Formatted at 2026-08-07 11:32:58 on dist-test-slave-wl2h"
server_key: "9bccf25cbbc856a944056346a5ce1bae"
server_key_iv: "6fcdcbcca7679173256351b218f13094"
server_key_version: "encryptionkey@0"
I20260807 11:32:58.391780 8355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:58.410714 8355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:58.415258 8355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:58.415560 8355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:58.435995 8355 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:42305
I20260807 11:32:58.436053 8431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:42305 every 8 connection(s)
I20260807 11:32:58.437513 8355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:32:58.441501 8432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:32:58.446136 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 8355
I20260807 11:32:58.446525 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:32:58.447369 24989 external_mini_cluster.cc:1428] Setting key b1e6d87691e27c836e2f496c8fe43184
I20260807 11:32:58.454479 8432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Bootstrap starting.
I20260807 11:32:58.458593 8432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Neither blocks nor log segments found. Creating new log.
I20260807 11:32:58.459956 8432 log.cc:824] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:32:58.463979 8432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: No bootstrap required, opened a new log
Aug 07 11:32:58 dist-test-slave-wl2h krb5kdc[8342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:58.475188 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:58.449533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.24.103.126:42305)
0807 11:32:58.450379 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:58.450390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:58.450433 (+ 43us) server_negotiation.cc:407] Connection header received
0807 11:32:58.452535 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:58.452559 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:58.452952 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:58.453319 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:58.455482 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:58.456982 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:58.459270 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:58.459966 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:58.460164 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:58.465558 (+ 5394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:58.465570 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:58.465595 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:58.465655 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:58.470052 (+ 4397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:58.470857 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:58.470871 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:58.470881 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:58.471133 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:58.471535 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:58.471540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:58.471544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:58.472389 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:58.472741 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:58.473142 (+ 401us) server_negotiation.cc:299] Negotiation successful
0807 11:32:58.473683 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":384,"threads_started":1}
I20260807 11:32:58.475344 8432 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:32:58.475826 8432 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:32:58.475980 8432 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f9fa9bf2fc43108bf298e006972b3e, State: Initialized, Role: FOLLOWER
I20260807 11:32:58.476671 8432 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:32:58.476958 8432 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:32:58.477137 8432 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:32:58.477334 8432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:32:58.479856 8432 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:32:58.480403 8432 leader_election.cc:302] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5f9fa9bf2fc43108bf298e006972b3e; no voters:
I20260807 11:32:58.481184 8432 leader_election.cc:288] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:32:58.481426 8438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:32:58.482816 8438 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 1 LEADER]: Becoming Leader. State: Replica: a5f9fa9bf2fc43108bf298e006972b3e, State: Running, Role: LEADER
I20260807 11:32:58.483489 8438 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:32:58.483924 8432 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:32:58.486639 8442 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5f9fa9bf2fc43108bf298e006972b3e. Latest consensus state: current_term: 1 leader_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } } }
I20260807 11:32:58.486766 8441 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } } }
I20260807 11:32:58.487107 8442 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: This master's current role is: LEADER
I20260807 11:32:58.487246 8441 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: This master's current role is: LEADER
I20260807 11:32:58.490327 8445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:32:58.496345 8445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:32:58.506666 8445 catalog_manager.cc:1347] Generated new cluster ID: 8dddcdf96f8048cd859f158387a4f592
I20260807 11:32:58.506804 8445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:32:58.526733 8445 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:32:58.528771 8445 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:32:58.541481 8445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Generated new TSK 0
I20260807 11:32:58.542605 8445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:32:58.623718 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42305
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:32:58.984701 8464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:58.985225 8464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:58.985347 8464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:58.993738 8464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:58.993852 8464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:58.993968 8464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:32:59.004603 8464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:59.005954 8464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:59.007398 8464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:59.021338 8471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:59.021291 8472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:59.021631 8475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:59.022666 8464 server_base.cc:1034] running on GCE node
I20260807 11:32:59.023451 8464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:59.024489 8464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:59.025799 8464 hybrid_clock.cc:648] HybridClock initialized: now 1786102379025735 us; error 47 us; skew 500 ppm
I20260807 11:32:59.026279 8464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:59 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102379, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:59.029928 8464 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:32:59.031679 8464 webserver.cc:460] Webserver started at http://127.24.103.65:41009/ using document root <none> and password file <none>
I20260807 11:32:59.032431 8464 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:59.032577 8464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:32:59.032928 8464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:32:59.036054 8464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "ee6182262691442daa97990f9959c3a6"
format_stamp: "Formatted at 2026-08-07 11:32:59 on dist-test-slave-wl2h"
server_key: "5698e7d737a4ce2a7a048da42932da3b"
server_key_iv: "0c31a7a212a97a7f6c41fd9c61d5d7c7"
server_key_version: "encryptionkey@0"
I20260807 11:32:59.036901 8464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "ee6182262691442daa97990f9959c3a6"
format_stamp: "Formatted at 2026-08-07 11:32:59 on dist-test-slave-wl2h"
server_key: "5698e7d737a4ce2a7a048da42932da3b"
server_key_iv: "0c31a7a212a97a7f6c41fd9c61d5d7c7"
server_key_version: "encryptionkey@0"
I20260807 11:32:59.042510 8464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:32:59.046348 8482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:59.047873 8464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 11:32:59.048060 8464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "ee6182262691442daa97990f9959c3a6"
format_stamp: "Formatted at 2026-08-07 11:32:59 on dist-test-slave-wl2h"
server_key: "5698e7d737a4ce2a7a048da42932da3b"
server_key_iv: "0c31a7a212a97a7f6c41fd9c61d5d7c7"
server_key_version: "encryptionkey@0"
I20260807 11:32:59.048277 8464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:59.070057 8464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:59.074707 8464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:59.075208 8464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:59.076308 8464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:32:59.077780 8464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:32:59.077908 8464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:59.078090 8464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:32:59.078176 8464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:59.115564 8464 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:45641
I20260807 11:32:59.115612 8607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:45641 every 8 connection(s)
I20260807 11:32:59.117094 8464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:32:59.121546 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 8464
I20260807 11:32:59.121795 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:32:59.122680 24989 external_mini_cluster.cc:1428] Setting key 7cb2cdfd1d8ee400502ea78e0318f011
Aug 07 11:32:59 dist-test-slave-wl2h krb5kdc[8342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102379, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:59.141719 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:59.119815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:40061 (local address 127.24.103.126:42305)
0807 11:32:59.120346 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:59.120350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:59.121246 (+ 896us) server_negotiation.cc:407] Connection header received
0807 11:32:59.123995 (+ 2749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:59.124000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:59.124099 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:59.124241 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:59.126162 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:59.126953 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:59.129505 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:59.129970 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:59.130093 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:59.134919 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:59.134925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:59.134929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:59.134966 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:59.137985 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:59.139131 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:59.139136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:59.139138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:59.139274 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:59.139834 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:59.139838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:59.139840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:59.140115 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:59.140242 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:59.141252 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0807 11:32:59.141436 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":317,"threads_started":1}
I20260807 11:32:59.143474 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:59.120282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:40061)
0807 11:32:59.121031 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:59.121084 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:59.123626 (+ 2542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:59.124440 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:59.124448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:59.125144 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:59.125941 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:59.125974 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:59.127178 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:59.127186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:59.129322 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:59.129341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:59.130216 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:59.130225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:59.130556 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:59.131446 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:59.131483 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:59.134638 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:59.138167 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:59.138176 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:59.138196 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:59.138908 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:59.139404 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:59.139412 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:59.139419 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:59.139691 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:59.140281 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:59.140292 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:59.140777 (+ 485us) client_negotiation.cc:769] Sending connection context
0807 11:32:59.141155 (+ 378us) client_negotiation.cc:240] Negotiation successful
0807 11:32:59.141822 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":290,"threads_started":1}
I20260807 11:32:59.145290 8608 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42305
I20260807 11:32:59.145656 8608 heartbeater.cc:464] Registering TS with master...
I20260807 11:32:59.146607 8608 heartbeater.cc:511] Master 127.24.103.126:42305 requested a full tablet report, sending...
I20260807 11:32:59.148866 8394 ts_manager.cc:194] Registered new tserver with Master: ee6182262691442daa97990f9959c3a6 (127.24.103.65:45641)
I20260807 11:32:59.150353 8394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:40061
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:32:59.194854 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42305
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:32:59.537880 8617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:32:59.538223 8617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:32:59.538403 8617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:32:59.551496 8617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:32:59.551728 8617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:32:59.551980 8617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:32:59.562788 8617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:32:59.564287 8617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:32:59.565649 8617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:32:59.579545 8626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:59.579613 8624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:32:59.579632 8622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:32:59.580848 8617 server_base.cc:1034] running on GCE node
I20260807 11:32:59.581473 8617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:32:59.582480 8617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:32:59.583698 8617 hybrid_clock.cc:648] HybridClock initialized: now 1786102379583629 us; error 52 us; skew 500 ppm
I20260807 11:32:59.584100 8617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:32:59 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102379, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:32:59.587828 8617 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:32:59.589490 8617 webserver.cc:460] Webserver started at http://127.24.103.66:38523/ using document root <none> and password file <none>
I20260807 11:32:59.590302 8617 fs_manager.cc:359] Metadata directory not provided
I20260807 11:32:59.590457 8617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:32:59.590850 8617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:32:59.594092 8617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "128a484d329d43728bd5447f6af1763f"
format_stamp: "Formatted at 2026-08-07 11:32:59 on dist-test-slave-wl2h"
server_key: "d4caa51690d8aa40ca2d5bccf79ccd82"
server_key_iv: "a0ae35e553bd1a4571c2edf1b44e2428"
server_key_version: "encryptionkey@0"
I20260807 11:32:59.594967 8617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "128a484d329d43728bd5447f6af1763f"
format_stamp: "Formatted at 2026-08-07 11:32:59 on dist-test-slave-wl2h"
server_key: "d4caa51690d8aa40ca2d5bccf79ccd82"
server_key_iv: "a0ae35e553bd1a4571c2edf1b44e2428"
server_key_version: "encryptionkey@0"
I20260807 11:32:59.601017 8617 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:32:59.604666 8635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:59.606253 8617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:32:59.606451 8617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "128a484d329d43728bd5447f6af1763f"
format_stamp: "Formatted at 2026-08-07 11:32:59 on dist-test-slave-wl2h"
server_key: "d4caa51690d8aa40ca2d5bccf79ccd82"
server_key_iv: "a0ae35e553bd1a4571c2edf1b44e2428"
server_key_version: "encryptionkey@0"
I20260807 11:32:59.606688 8617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:32:59.622049 8617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:32:59.626513 8617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:32:59.626857 8617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:32:59.627990 8617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:32:59.629396 8617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:32:59.629503 8617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:59.629658 8617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:32:59.629752 8617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:32:59.665899 8617 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:36899
I20260807 11:32:59.665947 8764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:36899 every 8 connection(s)
I20260807 11:32:59.667371 8617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:32:59.674841 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 8617
I20260807 11:32:59.675107 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:32:59.675642 24989 external_mini_cluster.cc:1428] Setting key fee08f3cbaf2806ae00771e6ddb6e7a8
Aug 07 11:32:59 dist-test-slave-wl2h krb5kdc[8342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102379, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:32:59.693440 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:59.670019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:39853 (local address 127.24.103.126:42305)
0807 11:32:59.670547 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0807 11:32:59.670565 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 11:32:59.671726 (+ 1161us) server_negotiation.cc:407] Connection header received
0807 11:32:59.673564 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:32:59.673569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:32:59.673648 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:32:59.673752 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:32:59.675614 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:59.676597 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:59.678971 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:32:59.679910 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:32:59.680097 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:59.684491 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:32:59.684502 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:32:59.684511 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:32:59.684567 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:32:59.689071 (+ 4504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:59.690287 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:59.690296 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:59.690302 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:59.690486 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:32:59.690967 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:32:59.690974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:32:59.690979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:32:59.691436 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:32:59.691599 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 11:32:59.692931 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0807 11:32:59.693149 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":225,"threads_started":1}
I20260807 11:32:59.695475 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:32:59.670579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:39853)
0807 11:32:59.671553 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0807 11:32:59.671596 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 11:32:59.673270 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:32:59.673980 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:32:59.673989 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:32:59.674611 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:32:59.675417 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:59.675437 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:59.676809 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:59.676820 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:59.678791 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:32:59.678805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:32:59.680254 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:32:59.680260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:32:59.680486 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:32:59.681306 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:32:59.681332 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:32:59.684165 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:32:59.689332 (+ 5167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:59.689342 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:59.689360 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:59.689941 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:59.690623 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:32:59.690627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:32:59.690629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:32:59.690843 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:32:59.691592 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:32:59.691600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:32:59.692405 (+ 805us) client_negotiation.cc:769] Sending connection context
0807 11:32:59.692785 (+ 380us) client_negotiation.cc:240] Negotiation successful
0807 11:32:59.693469 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":455,"threads_started":1}
I20260807 11:32:59.697322 8766 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42305
I20260807 11:32:59.697659 8766 heartbeater.cc:464] Registering TS with master...
I20260807 11:32:59.698484 8766 heartbeater.cc:511] Master 127.24.103.126:42305 requested a full tablet report, sending...
I20260807 11:32:59.700162 8394 ts_manager.cc:194] Registered new tserver with Master: 128a484d329d43728bd5447f6af1763f (127.24.103.66:36899)
I20260807 11:32:59.701061 8394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:39853
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:32:59.751894 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42305
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:33:00.104082 8772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:00.104411 8772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:00.104588 8772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:00.112298 8772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:00.112455 8772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:00.112645 8772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:33:00.122869 8772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:00.124181 8772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:00.125604 8772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:00.139204 8777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:00.139433 8778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:00.139696 8780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:00.140799 8772 server_base.cc:1034] running on GCE node
I20260807 11:33:00.141481 8772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:00.142469 8772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:00.143700 8772 hybrid_clock.cc:648] HybridClock initialized: now 1786102380143647 us; error 38 us; skew 500 ppm
I20260807 11:33:00.144055 8772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:00 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102380, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:00.147497 8772 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:33:00.149251 8772 webserver.cc:460] Webserver started at http://127.24.103.67:36477/ using document root <none> and password file <none>
I20260807 11:33:00.149948 8772 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:00.150068 8772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:00.150345 8772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:33:00.153898 8608 heartbeater.cc:503] Master 127.24.103.126:42305 was elected leader, sending a full tablet report...
I20260807 11:33:00.153741 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "79850c65db02473cb81f06f29889ae9e"
format_stamp: "Formatted at 2026-08-07 11:33:00 on dist-test-slave-wl2h"
server_key: "41a6a4b4e95077da1425f224168da4d0"
server_key_iv: "9c3b2d649255b18efc7493677970a2dd"
server_key_version: "encryptionkey@0"
I20260807 11:33:00.154812 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "79850c65db02473cb81f06f29889ae9e"
format_stamp: "Formatted at 2026-08-07 11:33:00 on dist-test-slave-wl2h"
server_key: "41a6a4b4e95077da1425f224168da4d0"
server_key_iv: "9c3b2d649255b18efc7493677970a2dd"
server_key_version: "encryptionkey@0"
I20260807 11:33:00.160769 8772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 11:33:00.164752 8787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:00.166203 8772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:33:00.166397 8772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "79850c65db02473cb81f06f29889ae9e"
format_stamp: "Formatted at 2026-08-07 11:33:00 on dist-test-slave-wl2h"
server_key: "41a6a4b4e95077da1425f224168da4d0"
server_key_iv: "9c3b2d649255b18efc7493677970a2dd"
server_key_version: "encryptionkey@0"
I20260807 11:33:00.166626 8772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:00.185029 8772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:00.189886 8772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:00.190214 8772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:00.191426 8772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:00.192965 8772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:00.193121 8772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:00.193290 8772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:00.193377 8772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:00.232178 8772 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:37847
I20260807 11:33:00.232229 8913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:37847 every 8 connection(s)
I20260807 11:33:00.233594 8772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:33:00.239801 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 8772
I20260807 11:33:00.240029 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:33:00.240660 24989 external_mini_cluster.cc:1428] Setting key 6b8c8e9ec37a5df03e0fd80e3ca78efa
Aug 07 11:33:00 dist-test-slave-wl2h krb5kdc[8342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102380, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:33:00.256304 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.236453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:49851 (local address 127.24.103.126:42305)
0807 11:33:00.237015 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:00.237019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:00.237838 (+ 819us) server_negotiation.cc:407] Connection header received
0807 11:33:00.239764 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:00.239770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:00.239875 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:00.240005 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:00.241976 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.242892 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.245246 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.245664 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.245773 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.249542 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:00.249547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:00.249551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:00.249583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:00.252657 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:00.253515 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:00.253520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:00.253523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:00.253653 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:00.254188 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:00.254192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:00.254194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:00.254475 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:00.254606 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:00.255610 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0807 11:33:00.255867 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":304,"threads_started":1}
I20260807 11:33:00.257325 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.236895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:49851)
0807 11:33:00.237664 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:00.237708 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:00.239477 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:00.240284 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:00.240296 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:00.241063 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:33:00.241727 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.241755 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.243222 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.243230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.245007 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.245029 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.245890 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.245895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.246163 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.246858 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:33:00.246881 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:33:00.249339 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:33:00.252837 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:00.252845 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:00.252858 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:00.253388 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:00.253856 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:00.253860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:00.253862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:00.254064 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:00.254624 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:33:00.254630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:33:00.254950 (+ 320us) client_negotiation.cc:769] Sending connection context
0807 11:33:00.255432 (+ 482us) client_negotiation.cc:240] Negotiation successful
0807 11:33:00.255972 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":310,"threads_started":1}
I20260807 11:33:00.258697 8914 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42305
I20260807 11:33:00.259002 8914 heartbeater.cc:464] Registering TS with master...
I20260807 11:33:00.259835 8914 heartbeater.cc:511] Master 127.24.103.126:42305 requested a full tablet report, sending...
I20260807 11:33:00.261545 8394 ts_manager.cc:194] Registered new tserver with Master: 79850c65db02473cb81f06f29889ae9e (127.24.103.67:37847)
I20260807 11:33:00.262528 8394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:49851
I20260807 11:33:00.269724 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:33:00.288903 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.275203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.24.103.126:42305)
0807 11:33:00.275471 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:00.275478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:00.275792 (+ 314us) server_negotiation.cc:407] Connection header received
0807 11:33:00.276004 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:00.276009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:00.276102 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:00.276213 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:00.277734 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.278491 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.279888 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.280326 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.280473 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.283090 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:00.283100 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:00.283104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:00.283137 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:00.286015 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:00.286750 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:00.286756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:00.286760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:00.286895 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:00.287338 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:00.287345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:00.287349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:00.287697 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:00.287958 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:00.288327 (+ 369us) server_negotiation.cc:299] Negotiation successful
0807 11:33:00.288564 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 11:33:00.293557 8394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:33:00.302425 8394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:33:00.337489 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.315033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.24.103.66:36899)
0807 11:33:00.328244 (+ 13211us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:00.328256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:00.328291 (+ 35us) server_negotiation.cc:407] Connection header received
0807 11:33:00.328408 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:00.328418 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:00.328703 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:00.328953 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:00.329935 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.331627 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.335312 (+ 3685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.336274 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.336433 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.336643 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:00.336932 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 11:33:00.337106 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13012,"thread_start_us":13509,"threads_started":1}
I20260807 11:33:00.337630 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.314753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:36899 (local address 127.0.0.1:42550)
0807 11:33:00.327296 (+ 12543us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:00.327361 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:00.327567 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:00.328953 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:00.328962 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:00.328994 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:00.329685 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.329703 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.331808 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.331816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.335130 (+ 3314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.335146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.336483 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.336490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.336669 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.336724 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 11:33:00.336893 (+ 169us) client_negotiation.cc:240] Negotiation successful
0807 11:33:00.337099 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12139,"mutex_wait_us":5,"thread_start_us":11929,"threads_started":1}
I20260807 11:33:00.338284 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.315067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37847 (local address 127.0.0.1:52536)
0807 11:33:00.328048 (+ 12981us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:00.328070 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:00.328207 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:00.329439 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:00.329445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:00.329476 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:00.329989 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.330004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.331925 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.331935 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.335173 (+ 3238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.335185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.337336 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.337343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.337562 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.337602 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 11:33:00.337702 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 11:33:00.337840 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12829,"mutex_wait_us":66,"thread_start_us":12673,"threads_started":1}
I20260807 11:33:00.338774 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.315507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.24.103.67:37847)
0807 11:33:00.326639 (+ 11132us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:00.326654 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:00.328359 (+ 1705us) server_negotiation.cc:407] Connection header received
0807 11:33:00.328471 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:00.328479 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:00.328715 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:00.328937 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:00.330219 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.331780 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.335367 (+ 3587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.336796 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.336930 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.337180 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:00.338262 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0807 11:33:00.338435 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10879,"thread_start_us":10642,"threads_started":1}
I20260807 11:33:00.341770 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.329727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.0.0.1:39906)
0807 11:33:00.333243 (+ 3516us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:00.333262 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:00.333400 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:00.334156 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:00.334161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:00.334187 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:00.334690 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.334731 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.336754 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.336760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.339117 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.339134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.341028 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.341034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.341205 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.341238 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 11:33:00.341344 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 11:33:00.341464 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3381,"thread_start_us":3259,"threads_started":1}
I20260807 11:33:00.341898 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.328526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.24.103.65:45641)
0807 11:33:00.332966 (+ 4440us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:00.332978 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:00.333546 (+ 568us) server_negotiation.cc:407] Connection header received
0807 11:33:00.333676 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:00.333684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:00.333934 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:00.334140 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:00.334943 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.336565 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.339281 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.340797 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.340948 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.341168 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:00.341411 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 11:33:00.341567 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4262,"thread_start_us":4085,"threads_started":1}
I20260807 11:33:00.342551 8688 tablet_service.cc:1467] Processing CreateTablet for tablet c245dc9e36e34f508d4c1343e0bee884 (DEFAULT_TABLE table=test-table [id=a4747d1ed85c4a0bb9a3ff3eb3a818b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:00.342592 8841 tablet_service.cc:1467] Processing CreateTablet for tablet c245dc9e36e34f508d4c1343e0bee884 (DEFAULT_TABLE table=test-table [id=a4747d1ed85c4a0bb9a3ff3eb3a818b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:00.343911 8841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c245dc9e36e34f508d4c1343e0bee884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:00.343986 8688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c245dc9e36e34f508d4c1343e0bee884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:00.345631 8532 tablet_service.cc:1467] Processing CreateTablet for tablet c245dc9e36e34f508d4c1343e0bee884 (DEFAULT_TABLE table=test-table [id=a4747d1ed85c4a0bb9a3ff3eb3a818b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:00.347084 8532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c245dc9e36e34f508d4c1343e0bee884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:00.355042 8938 tablet_bootstrap.cc:492] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: Bootstrap starting.
I20260807 11:33:00.359475 8938 tablet_bootstrap.cc:654] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:00.360651 8938 log.cc:824] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:00.361202 8939 tablet_bootstrap.cc:492] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: Bootstrap starting.
I20260807 11:33:00.366045 8939 tablet_bootstrap.cc:654] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:00.367861 8939 log.cc:824] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:00.367947 8940 tablet_bootstrap.cc:492] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Bootstrap starting.
I20260807 11:33:00.371021 8938 tablet_bootstrap.cc:492] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: No bootstrap required, opened a new log
I20260807 11:33:00.371409 8938 ts_tablet_manager.cc:1397] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20260807 11:33:00.371443 8939 tablet_bootstrap.cc:492] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: No bootstrap required, opened a new log
I20260807 11:33:00.371717 8939 ts_tablet_manager.cc:1397] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260807 11:33:00.371742 8940 tablet_bootstrap.cc:654] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:00.373042 8940 log.cc:824] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:00.376641 8940 tablet_bootstrap.cc:492] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: No bootstrap required, opened a new log
I20260807 11:33:00.376900 8940 ts_tablet_manager.cc:1397] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:33:00.379812 8938 raft_consensus.cc:348] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:00.380285 8938 raft_consensus.cc:374] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:00.380393 8938 raft_consensus.cc:729] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79850c65db02473cb81f06f29889ae9e, State: Initialized, Role: FOLLOWER
I20260807 11:33:00.381065 8938 consensus_queue.cc:260] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:00.382215 8914 heartbeater.cc:503] Master 127.24.103.126:42305 was elected leader, sending a full tablet report...
I20260807 11:33:00.382066 8939 raft_consensus.cc:348] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:00.382496 8939 raft_consensus.cc:374] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:00.382606 8939 raft_consensus.cc:729] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128a484d329d43728bd5447f6af1763f, State: Initialized, Role: FOLLOWER
I20260807 11:33:00.383080 8938 ts_tablet_manager.cc:1428] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260807 11:33:00.383217 8939 consensus_queue.cc:260] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:00.384438 8766 heartbeater.cc:503] Master 127.24.103.126:42305 was elected leader, sending a full tablet report...
I20260807 11:33:00.385015 8939 ts_tablet_manager.cc:1428] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260807 11:33:00.388926 8940 raft_consensus.cc:348] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:00.389395 8940 raft_consensus.cc:374] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:00.389501 8940 raft_consensus.cc:729] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6182262691442daa97990f9959c3a6, State: Initialized, Role: FOLLOWER
I20260807 11:33:00.389966 8940 consensus_queue.cc:260] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:00.391544 8940 ts_tablet_manager.cc:1428] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260807 11:33:00.419490 8767 tablet.cc:2369] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:33:00.485525 8915 tablet.cc:2369] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:33:00.619812 8609 tablet.cc:2369] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:33:00.677091 8944 raft_consensus.cc:482] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:33:00.677419 8944 raft_consensus.cc:504] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:00.679256 8944 leader_election.cc:288] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128a484d329d43728bd5447f6af1763f (127.24.103.66:36899), ee6182262691442daa97990f9959c3a6 (127.24.103.65:45641)
I20260807 11:33:00.686057 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.679693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:36899 (local address 127.24.103.67:42457)
0807 11:33:00.680062 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:00.680078 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:00.680254 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:00.680615 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:00.680620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:00.680659 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:00.680836 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.680844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.681845 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.681850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.684064 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.684080 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.685451 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.685456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.685603 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.685635 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 11:33:00.685717 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 11:33:00.685820 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260807 11:33:00.686167 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.679705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:42457 (local address 127.24.103.66:36899)
0807 11:33:00.679923 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:00.679929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:00.680203 (+ 274us) server_negotiation.cc:407] Connection header received
0807 11:33:00.680410 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:00.680421 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:00.680480 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:00.680584 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:00.680972 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.681725 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.684247 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.685300 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.685468 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.685551 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:00.685766 (+ 215us) server_negotiation.cc:299] Negotiation successful
0807 11:33:00.685906 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:33:00.687430 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.679709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:45569 (local address 127.24.103.65:45641)
0807 11:33:00.679982 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:00.679988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:00.680590 (+ 602us) server_negotiation.cc:407] Connection header received
0807 11:33:00.680862 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:00.680868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:00.680936 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:00.681034 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:00.681799 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.682732 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.685182 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.686471 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.686650 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.686724 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:00.687050 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 11:33:00.687186 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260807 11:33:00.687544 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.679709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.24.103.67:45569)
0807 11:33:00.680474 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:00.680489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:00.680706 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:00.681047 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:00.681051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:00.681073 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:00.681588 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.681603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.682910 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.682916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.684937 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:00.684950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.686699 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.686705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:00.686864 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.686887 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 11:33:00.686992 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 11:33:00.687142 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":90,"spinlock_wait_cycles":44800,"thread_start_us":367,"threads_started":1}
I20260807 11:33:00.687655 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c245dc9e36e34f508d4c1343e0bee884" candidate_uuid: "79850c65db02473cb81f06f29889ae9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128a484d329d43728bd5447f6af1763f" is_pre_election: true
I20260807 11:33:00.688362 8711 raft_consensus.cc:2454] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79850c65db02473cb81f06f29889ae9e in term 0.
I20260807 11:33:00.688673 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c245dc9e36e34f508d4c1343e0bee884" candidate_uuid: "79850c65db02473cb81f06f29889ae9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6182262691442daa97990f9959c3a6" is_pre_election: true
I20260807 11:33:00.689312 8556 raft_consensus.cc:2454] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79850c65db02473cb81f06f29889ae9e in term 0.
I20260807 11:33:00.689339 8788 leader_election.cc:302] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128a484d329d43728bd5447f6af1763f, 79850c65db02473cb81f06f29889ae9e; no voters:
I20260807 11:33:00.689848 8944 raft_consensus.cc:2781] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:33:00.690088 8944 raft_consensus.cc:482] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:33:00.690251 8944 raft_consensus.cc:3037] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:00.692929 8944 raft_consensus.cc:504] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:00.693636 8944 leader_election.cc:288] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [CANDIDATE]: Term 1 election: Requested vote from peers 128a484d329d43728bd5447f6af1763f (127.24.103.66:36899), ee6182262691442daa97990f9959c3a6 (127.24.103.65:45641)
I20260807 11:33:00.694119 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c245dc9e36e34f508d4c1343e0bee884" candidate_uuid: "79850c65db02473cb81f06f29889ae9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128a484d329d43728bd5447f6af1763f"
I20260807 11:33:00.694172 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c245dc9e36e34f508d4c1343e0bee884" candidate_uuid: "79850c65db02473cb81f06f29889ae9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6182262691442daa97990f9959c3a6"
I20260807 11:33:00.694332 8711 raft_consensus.cc:3037] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:00.694438 8556 raft_consensus.cc:3037] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:00.697013 8711 raft_consensus.cc:2454] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79850c65db02473cb81f06f29889ae9e in term 1.
I20260807 11:33:00.697352 8556 raft_consensus.cc:2454] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79850c65db02473cb81f06f29889ae9e in term 1.
I20260807 11:33:00.697446 8788 leader_election.cc:302] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128a484d329d43728bd5447f6af1763f, 79850c65db02473cb81f06f29889ae9e; no voters:
I20260807 11:33:00.697777 8944 raft_consensus.cc:2781] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:33:00.698277 8944 raft_consensus.cc:686] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 1 LEADER]: Becoming Leader. State: Replica: 79850c65db02473cb81f06f29889ae9e, State: Running, Role: LEADER
I20260807 11:33:00.698796 8944 consensus_queue.cc:237] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:00.703892 8394 catalog_manager.cc:5495] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 79850c65db02473cb81f06f29889ae9e (127.24.103.67). New cstate: current_term: 1 leader_uuid: "79850c65db02473cb81f06f29889ae9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } health_report { overall_health: UNKNOWN } } }
I20260807 11:33:00.785771 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.779799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.24.103.67:37847)
0807 11:33:00.780034 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:00.780040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:00.780064 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:33:00.780319 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:00.780323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:00.780420 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:00.780568 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:33:00.780956 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.781748 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.783357 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.783774 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.783880 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.784498 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:33:00.785035 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:33:00.785304 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:00.785393 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 11:33:00.785484 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260807 11:33:00.794711 8711 raft_consensus.cc:1261] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f [term 1 FOLLOWER]: Refusing update from remote peer 79850c65db02473cb81f06f29889ae9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:33:00.794716 8556 raft_consensus.cc:1261] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Refusing update from remote peer 79850c65db02473cb81f06f29889ae9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:33:00.795796 8944 consensus_queue.cc:1035] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:33:00.796159 8948 consensus_queue.cc:1035] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:33:00.824206 8394 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:33:00.824506 8394 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:33:00.828290 8394 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Sending DeleteTablet for 3 replicas of tablet c245dc9e36e34f508d4c1343e0bee884
I20260807 11:33:00.829569 8688 tablet_service.cc:1514] Processing DeleteTablet for tablet c245dc9e36e34f508d4c1343e0bee884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:00 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:42550
I20260807 11:33:00.829675 8532 tablet_service.cc:1514] Processing DeleteTablet for tablet c245dc9e36e34f508d4c1343e0bee884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:00 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:39906
I20260807 11:33:00.830060 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 8464
I20260807 11:33:00.830623 8841 tablet_service.cc:1514] Processing DeleteTablet for tablet c245dc9e36e34f508d4c1343e0bee884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:00 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:52536
Aug 07 11:33:00 dist-test-slave-wl2h krb5kdc[8342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:33:00.856002 8964 tablet_replica.cc:330] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: stopping tablet replica
I20260807 11:33:00.856592 8964 raft_consensus.cc:2229] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:33:00.857548 8964 raft_consensus.cc:2258] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:00.858471 8963 tablet_replica.cc:330] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: stopping tablet replica
I20260807 11:33:00.858886 8963 raft_consensus.cc:2229] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:33:00.859431 8963 raft_consensus.cc:2258] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:00.862038 8964 ts_tablet_manager.cc:1905] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:00.863535 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.831680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.24.103.65:45641)
0807 11:33:00.831919 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:00.831928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:00.839711 (+ 7783us) server_negotiation.cc:407] Connection header received
0807 11:33:00.839977 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:00.839986 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:00.840106 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:00.840194 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:00.842737 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.844195 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.846708 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:00.847583 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:00.847717 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:00.854810 (+ 7093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:00.854835 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:00.854862 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:00.854925 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:00.859289 (+ 4364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:00.860333 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:00.860343 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:00.860348 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:00.860528 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:00.861042 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:00.861047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:00.861050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:00.861508 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:00.861816 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:00.862486 (+ 670us) server_negotiation.cc:299] Negotiation successful
0807 11:33:00.862759 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:33:00.865083 8602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:00.867883 8962 tablet_replica.cc:330] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: stopping tablet replica
I20260807 11:33:00.868474 8962 raft_consensus.cc:2229] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:33:00.869139 8962 raft_consensus.cc:2258] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:00.874755 8964 ts_tablet_manager.cc:1918] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:00.875000 8964 log.cc:1196] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/c245dc9e36e34f508d4c1343e0bee884
I20260807 11:33:00.875806 8964 ts_tablet_manager.cc:1939] T c245dc9e36e34f508d4c1343e0bee884 P 79850c65db02473cb81f06f29889ae9e: Deleting consensus metadata
I20260807 11:33:00.878152 8379 catalog_manager.cc:4841] TS 79850c65db02473cb81f06f29889ae9e (127.24.103.67:37847): tablet c245dc9e36e34f508d4c1343e0bee884 (table test-table [id=a4747d1ed85c4a0bb9a3ff3eb3a818b1]) successfully deleted
I20260807 11:33:00.880100 8962 ts_tablet_manager.cc:1905] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:00.888556 8962 ts_tablet_manager.cc:1918] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:00.888773 8962 log.cc:1196] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/c245dc9e36e34f508d4c1343e0bee884
I20260807 11:33:00.889321 8962 ts_tablet_manager.cc:1939] T c245dc9e36e34f508d4c1343e0bee884 P 128a484d329d43728bd5447f6af1763f: Deleting consensus metadata
I20260807 11:33:00.891474 8377 catalog_manager.cc:4841] TS 128a484d329d43728bd5447f6af1763f (127.24.103.66:36899): tablet c245dc9e36e34f508d4c1343e0bee884 (table test-table [id=a4747d1ed85c4a0bb9a3ff3eb3a818b1]) successfully deleted
I20260807 11:33:00.970973 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 8464
W20260807 11:33:00.995283 8379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:45641 (error 108)
I20260807 11:33:00.996222 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 8617
I20260807 11:33:00.997987 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.997019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.0.0.1:39922)
0807 11:33:00.997628 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:00.997747 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":374,"threads_started":1}
W20260807 11:33:00.998301 8379 catalog_manager.cc:4584] TS ee6182262691442daa97990f9959c3a6 (127.24.103.65:45641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c245dc9e36e34f508d4c1343e0bee884: Network error: Client connection negotiation failed: client connection to 127.24.103.65:45641: connect: Connection refused (error 111)
Aug 07 11:33:01 dist-test-slave-wl2h krb5kdc[8342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:33:01.015372 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:00.997026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.24.103.66:36899)
0807 11:33:00.997246 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:00.997255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:00.997612 (+ 357us) server_negotiation.cc:407] Connection header received
0807 11:33:00.997692 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:00.997698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:00.997783 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:00.997891 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:00.999547 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:01.000785 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:01.002161 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:01.002804 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:01.002944 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:01.007853 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:01.007875 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:01.007897 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:01.007954 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:01.012108 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:01.012948 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:01.012959 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:01.012964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:01.013140 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:01.013505 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:01.013510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:01.013514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:01.014037 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:01.014355 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:01.014665 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 11:33:01.014936 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:33:01.016218 8760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:01.052203 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:01.051595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.0.0.1:39926)
0807 11:33:01.051864 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:01.051995 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260807 11:33:01.053539 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:01.053093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.0.0.1:39934)
0807 11:33:01.053273 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:01.053365 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 11:33:01.110803 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 8617
I20260807 11:33:01.115401 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:01.114831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.0.0.1:39944)
0807 11:33:01.115137 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:01.115238 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260807 11:33:01.116528 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:01.116169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.0.0.1:39948)
0807 11:33:01.116320 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:01.116397 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 11:33:01.139578 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 8772
Aug 07 11:33:01 dist-test-slave-wl2h krb5kdc[8342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:33:01.159219 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:01.140420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.24.103.67:37847)
0807 11:33:01.140603 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:01.140607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:01.140699 (+ 92us) server_negotiation.cc:407] Connection header received
0807 11:33:01.140897 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:01.140901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:01.140990 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:01.141083 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:01.142910 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:01.144109 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:01.146150 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:01.146656 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:01.146764 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:01.152731 (+ 5967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:01.152747 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:01.152764 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:01.152816 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:01.156095 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:01.156880 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:01.156888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:01.156891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:01.157056 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:01.157463 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:01.157469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:01.157472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:01.157854 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:01.158188 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:01.158506 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 11:33:01.158753 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:33:01.160153 8908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:01.197719 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:01.197029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.0.0.1:39950)
0807 11:33:01.197418 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:01.197520 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237}
I20260807 11:33:01.199172 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:01.198483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.0.0.1:39962)
0807 11:33:01.198716 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:01.198804 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260807 11:33:01.287448 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 8772
I20260807 11:33:01.314311 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 8355
I20260807 11:33:01.315723 8427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:01.433689 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:01.432828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.0.0.1:39972)
0807 11:33:01.433186 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:01.433428 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260807 11:33:01.433728 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 8355
I20260807 11:33:01.460803 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42305
--webserver_interface=127.24.103.126
--webserver_port=34591
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:33:01.814378 8975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:01.814771 8975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:01.814953 8975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:01.822675 8975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:33:01.822834 8975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:01.822949 8975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:01.823060 8975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:33:01.823141 8975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:33:01.833446 8975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=34591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:01.834928 8975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:01.836331 8975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:01.863854 8983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:01.863864 8982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:01.864559 8985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:01.865000 8975 server_base.cc:1034] running on GCE node
I20260807 11:33:01.865922 8975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:01.867756 8975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:01.868983 8975 hybrid_clock.cc:648] HybridClock initialized: now 1786102381868932 us; error 42 us; skew 500 ppm
I20260807 11:33:01.869375 8975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:01 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102381, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:01.877586 8975 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:33:01.879607 8975 webserver.cc:460] Webserver started at http://127.24.103.126:34591/ using document root <none> and password file <none>
I20260807 11:33:01.880476 8975 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:01.880637 8975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:01.887091 8975 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 11:33:01.890403 8993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:01.892153 8975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:33:01.892388 8975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "a5f9fa9bf2fc43108bf298e006972b3e"
format_stamp: "Formatted at 2026-08-07 11:32:58 on dist-test-slave-wl2h"
server_key: "9bccf25cbbc856a944056346a5ce1bae"
server_key_iv: "6fcdcbcca7679173256351b218f13094"
server_key_version: "encryptionkey@0"
I20260807 11:33:01.893610 8975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:01.913188 8975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:01.918165 8975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:01.918516 8975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:01.939877 8975 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:42305
I20260807 11:33:01.939904 9051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:42305 every 8 connection(s)
I20260807 11:33:01.941541 8975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:33:01.943300 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 8975
I20260807 11:33:01.944386 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:45641
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:42305
--webserver_port=41009
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:33:01.952370 9054 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:33:01.957746 9054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Bootstrap starting.
I20260807 11:33:01.979662 9054 log.cc:824] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:01.993819 9054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:33:01.994457 9054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Bootstrap complete.
I20260807 11:33:02.005111 9054 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:33:02.006050 9054 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f9fa9bf2fc43108bf298e006972b3e, State: Initialized, Role: FOLLOWER
I20260807 11:33:02.006845 9054 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:33:02.007153 9054 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:33:02.007315 9054 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:33:02.007515 9054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:33:02.011858 9054 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:33:02.012517 9054 leader_election.cc:302] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5f9fa9bf2fc43108bf298e006972b3e; no voters:
I20260807 11:33:02.013213 9054 leader_election.cc:288] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 11:33:02.013381 9059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:33:02.014691 9059 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 2 LEADER]: Becoming Leader. State: Replica: a5f9fa9bf2fc43108bf298e006972b3e, State: Running, Role: LEADER
I20260807 11:33:02.015429 9059 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:33:02.015905 9054 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:33:02.018561 9060 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } } }
I20260807 11:33:02.018565 9061 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5f9fa9bf2fc43108bf298e006972b3e. Latest consensus state: current_term: 2 leader_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } } }
I20260807 11:33:02.019225 9060 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: This master's current role is: LEADER
I20260807 11:33:02.019249 9061 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: This master's current role is: LEADER
I20260807 11:33:02.030910 9065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:33:02.041036 9065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:33:02.043972 9065 catalog_manager.cc:1259] Loaded cluster ID: 8dddcdf96f8048cd859f158387a4f592
I20260807 11:33:02.044180 9065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:33:02.048285 9065 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 11:33:02.059532 9077 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 11:33:02.064281 9065 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 11:33:02.071933 9065 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:33:02.073549 9065 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Loaded TSK: 0
I20260807 11:33:02.074833 9065 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 11:33:02.342849 9057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:02.343319 9057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:02.343488 9057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:02.350984 9057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:02.351181 9057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:02.351384 9057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:33:02.361598 9057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:45641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=41009
--tserver_master_addrs=127.24.103.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:02.362979 9057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:02.364362 9057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:02.379319 9085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:02.380119 9086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:02.380364 9088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:02.380813 9057 server_base.cc:1034] running on GCE node
I20260807 11:33:02.381618 9057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:02.382933 9057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:02.384223 9057 hybrid_clock.cc:648] HybridClock initialized: now 1786102382384131 us; error 80 us; skew 500 ppm
I20260807 11:33:02.384641 9057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:02 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102382, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:02.389055 9057 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:33:02.391146 9057 webserver.cc:460] Webserver started at http://127.24.103.65:41009/ using document root <none> and password file <none>
I20260807 11:33:02.392078 9057 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:02.392244 9057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:02.400161 9057 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:33:02.404285 9096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:02.406195 9057 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260807 11:33:02.406399 9057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "ee6182262691442daa97990f9959c3a6"
format_stamp: "Formatted at 2026-08-07 11:32:59 on dist-test-slave-wl2h"
server_key: "5698e7d737a4ce2a7a048da42932da3b"
server_key_iv: "0c31a7a212a97a7f6c41fd9c61d5d7c7"
server_key_version: "encryptionkey@0"
I20260807 11:33:02.408022 9057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:02.435159 9057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:02.441138 9057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:02.441468 9057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:02.442759 9057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:02.445123 9104 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 11:33:02.451594 9057 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 11:33:02.451757 9057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260807 11:33:02.451905 9057 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 11:33:02.456960 9057 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 11:33:02.457072 9057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20260807 11:33:02.457527 9104 tablet_bootstrap.cc:492] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Bootstrap starting.
I20260807 11:33:02.467227 9104 log.cc:824] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:02.484524 9104 tablet_bootstrap.cc:492] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:33:02.485668 9104 tablet_bootstrap.cc:492] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Bootstrap complete.
I20260807 11:33:02.486469 9104 ts_tablet_manager.cc:1397] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.009s
I20260807 11:33:02.498281 9104 raft_consensus.cc:348] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:02.500147 9104 raft_consensus.cc:729] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6182262691442daa97990f9959c3a6, State: Initialized, Role: FOLLOWER
I20260807 11:33:02.500877 9104 consensus_queue.cc:260] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:02.502439 9104 ts_tablet_manager.cc:1428] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
I20260807 11:33:02.507035 9057 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:45641
I20260807 11:33:02.507153 9228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:45641 every 8 connection(s)
I20260807 11:33:02.508392 9057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:33:02.509282 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 9057
I20260807 11:33:02.509943 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:36899
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:42305
--webserver_port=38523
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:33:02 dist-test-slave-wl2h krb5kdc[8342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102382, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:33:02.533453 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:02.512999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:45387 (local address 127.24.103.126:42305)
0807 11:33:02.514235 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:02.514246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:02.514283 (+ 37us) server_negotiation.cc:407] Connection header received
0807 11:33:02.514543 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:02.514556 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:02.514853 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:02.515327 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:02.517198 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:02.518468 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:02.520103 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:02.520856 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:02.521065 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:02.525057 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:02.525064 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:02.525080 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:02.525117 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:02.528191 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:02.529321 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:02.529329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:02.529334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:02.529561 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:02.530141 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:02.530145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:02.530147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:02.530634 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:02.530788 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:02.531938 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0807 11:33:02.532447 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":873,"threads_started":1}
I20260807 11:33:02.534020 9231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:02.512052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:45387)
0807 11:33:02.512920 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:02.512976 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:02.513502 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:02.515449 (+ 1947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:02.515460 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:02.516253 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:33:02.516906 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:02.516926 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:02.518636 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:02.518642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:02.519938 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:02.519951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:02.521063 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:02.521068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:02.521278 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:02.522050 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:33:02.522078 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:33:02.524751 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:33:02.528356 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:02.528370 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:02.528389 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:02.529116 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:02.529717 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:02.529725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:02.529731 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:02.529994 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:02.530782 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:33:02.530793 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:33:02.531286 (+ 493us) client_negotiation.cc:769] Sending connection context
0807 11:33:02.531712 (+ 426us) client_negotiation.cc:240] Negotiation successful
0807 11:33:02.532339 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":1038,"threads_started":1}
I20260807 11:33:02.537204 9229 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42305
I20260807 11:33:02.537601 9229 heartbeater.cc:464] Registering TS with master...
I20260807 11:33:02.538582 9229 heartbeater.cc:511] Master 127.24.103.126:42305 requested a full tablet report, sending...
I20260807 11:33:02.542614 9013 ts_manager.cc:194] Registered new tserver with Master: ee6182262691442daa97990f9959c3a6 (127.24.103.65:45641)
I20260807 11:33:02.547772 9013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:45387
I20260807 11:33:02.558703 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:02.546951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.24.103.65:45641)
0807 11:33:02.551773 (+ 4822us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:02.551779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:02.551809 (+ 30us) server_negotiation.cc:407] Connection header received
0807 11:33:02.551887 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:02.551893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:02.552104 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:02.552336 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:02.552987 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:02.554144 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:02.556251 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:02.557264 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:02.557456 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:02.557730 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:02.558083 (+ 353us) server_negotiation.cc:299] Negotiation successful
0807 11:33:02.558255 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4651,"thread_start_us":4462,"threads_started":1}
I20260807 11:33:02.558682 9234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:02.546724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.0.0.1:39978)
0807 11:33:02.549877 (+ 3153us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:02.549938 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:02.550173 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:02.552356 (+ 2183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:02.552364 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:02.552404 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:02.552781 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:02.552792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:02.554303 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:02.554311 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:02.556056 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:02.556073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:02.557539 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:02.557546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:02.557811 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:02.557879 (+ 68us) client_negotiation.cc:769] Sending connection context
0807 11:33:02.558026 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 11:33:02.558263 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2817,"thread_start_us":3696,"threads_started":1}
I20260807 11:33:02.560586 9155 tablet_service.cc:1514] Processing DeleteTablet for tablet c245dc9e36e34f508d4c1343e0bee884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:00 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:39978
I20260807 11:33:02.562907 9237 tablet_replica.cc:330] stopping tablet replica
I20260807 11:33:02.563370 9237 raft_consensus.cc:2229] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:33:02.563820 9237 raft_consensus.cc:2258] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:02.565320 9237 ts_tablet_manager.cc:1905] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:02.576200 9237 ts_tablet_manager.cc:1918] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:02.576423 9237 log.cc:1196] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/c245dc9e36e34f508d4c1343e0bee884
I20260807 11:33:02.577030 9237 ts_tablet_manager.cc:1939] T c245dc9e36e34f508d4c1343e0bee884 P ee6182262691442daa97990f9959c3a6: Deleting consensus metadata
I20260807 11:33:02.580060 8996 catalog_manager.cc:4841] TS ee6182262691442daa97990f9959c3a6 (127.24.103.65:45641): tablet c245dc9e36e34f508d4c1343e0bee884 (table test-table [id=a4747d1ed85c4a0bb9a3ff3eb3a818b1]) successfully deleted
W20260807 11:33:02.869408 9233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:02.869738 9233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:02.869908 9233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:02.877416 9233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:02.877576 9233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:02.877794 9233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:33:02.888399 9233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:36899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=38523
--tserver_master_addrs=127.24.103.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:02.889863 9233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:02.891357 9233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:02.906672 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:02.906663 9247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:02.906801 9244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:02.907641 9233 server_base.cc:1034] running on GCE node
I20260807 11:33:02.908397 9233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:02.909524 9233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:02.910761 9233 hybrid_clock.cc:648] HybridClock initialized: now 1786102382910693 us; error 66 us; skew 500 ppm
I20260807 11:33:02.911254 9233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:02 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102382, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:02.914927 9233 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:33:02.916594 9233 webserver.cc:460] Webserver started at http://127.24.103.66:38523/ using document root <none> and password file <none>
I20260807 11:33:02.917368 9233 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:02.917527 9233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:02.923662 9233 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:33:02.927300 9255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:02.928876 9233 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 11:33:02.929100 9233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "128a484d329d43728bd5447f6af1763f"
format_stamp: "Formatted at 2026-08-07 11:32:59 on dist-test-slave-wl2h"
server_key: "d4caa51690d8aa40ca2d5bccf79ccd82"
server_key_iv: "a0ae35e553bd1a4571c2edf1b44e2428"
server_key_version: "encryptionkey@0"
I20260807 11:33:02.930197 9233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:02.950084 9233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:02.954543 9233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:02.954842 9233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:02.955967 9233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:02.957341 9233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:02.957471 9233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:02.957615 9233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:02.957713 9233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:02.995554 9233 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:36899
I20260807 11:33:02.995610 9397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:36899 every 8 connection(s)
I20260807 11:33:02.997038 9233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:33:03.004350 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 9233
I20260807 11:33:03.005043 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:37847
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:42305
--webserver_port=36477
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:33:03 dist-test-slave-wl2h krb5kdc[8342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102382, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:33:03.020349 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.000001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:33351 (local address 127.24.103.126:42305)
0807 11:33:03.000237 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:03.000243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:03.001426 (+ 1183us) server_negotiation.cc:407] Connection header received
0807 11:33:03.003462 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:03.003467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:03.003561 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:03.003658 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:03.005271 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.005985 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.007686 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.008124 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.008272 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.012255 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:03.012265 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:03.012273 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:03.012325 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:03.016435 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:03.017666 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:03.017671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:03.017673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:03.017813 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:03.018376 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:03.018379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:03.018381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:03.018673 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:03.018790 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:03.019862 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0807 11:33:03.020073 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 11:33:03.022162 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.000489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:33351)
0807 11:33:03.001256 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:03.001305 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:03.003146 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:03.003887 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:03.003896 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:03.004454 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:33:03.005067 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.005084 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.006173 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.006178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.007504 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.007518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.008386 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.008393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.008622 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.009332 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:33:03.009360 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:33:03.011934 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:33:03.016659 (+ 4725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:03.016681 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:03.016705 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:03.017528 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:03.017946 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:03.017952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:03.017959 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:03.018248 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:03.018817 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:33:03.018829 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:33:03.019410 (+ 581us) client_negotiation.cc:769] Sending connection context
0807 11:33:03.019763 (+ 353us) client_negotiation.cc:240] Negotiation successful
0807 11:33:03.020429 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":292,"threads_started":1}
I20260807 11:33:03.023891 9398 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42305
I20260807 11:33:03.024317 9398 heartbeater.cc:464] Registering TS with master...
I20260807 11:33:03.025125 9398 heartbeater.cc:511] Master 127.24.103.126:42305 requested a full tablet report, sending...
I20260807 11:33:03.026759 9013 ts_manager.cc:194] Registered new tserver with Master: 128a484d329d43728bd5447f6af1763f (127.24.103.66:36899)
I20260807 11:33:03.027680 9013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:33351
W20260807 11:33:03.354293 9401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:03.354614 9401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:03.354795 9401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:03.362232 9401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:03.362383 9401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:03.362582 9401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:33:03.372608 9401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:37847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=36477
--tserver_master_addrs=127.24.103.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:03.373939 9401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:03.375366 9401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:03.393080 9412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:03.393931 9409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:03.394486 9408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:03.395454 9401 server_base.cc:1034] running on GCE node
I20260807 11:33:03.396205 9401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:03.397614 9401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:03.399039 9401 hybrid_clock.cc:648] HybridClock initialized: now 1786102383398920 us; error 78 us; skew 500 ppm
I20260807 11:33:03.399557 9401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:03 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102383, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:03.403915 9401 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:33:03.406209 9401 webserver.cc:460] Webserver started at http://127.24.103.67:36477/ using document root <none> and password file <none>
I20260807 11:33:03.407272 9401 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:03.407466 9401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:03.415905 9401 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:33:03.418792 9421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:03.420369 9401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:33:03.420522 9401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "79850c65db02473cb81f06f29889ae9e"
format_stamp: "Formatted at 2026-08-07 11:33:00 on dist-test-slave-wl2h"
server_key: "41a6a4b4e95077da1425f224168da4d0"
server_key_iv: "9c3b2d649255b18efc7493677970a2dd"
server_key_version: "encryptionkey@0"
I20260807 11:33:03.421495 9401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:03.446034 9401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:03.451591 9401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:03.451929 9401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:03.453313 9401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:03.455139 9401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:03.455273 9401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:03.455394 9401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:03.455484 9401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:03.494799 9401 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:37847
I20260807 11:33:03.494943 9550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:37847 every 8 connection(s)
I20260807 11:33:03.496479 9401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:33:03.501116 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 9401
Aug 07 11:33:03 dist-test-slave-wl2h krb5kdc[8342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102383, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:33:03.526029 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.502239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.24.103.126:42305)
0807 11:33:03.502799 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:03.502807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:03.507965 (+ 5158us) server_negotiation.cc:407] Connection header received
0807 11:33:03.509405 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:03.509410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:03.509477 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:03.509574 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:03.511731 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.512967 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.515256 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.515923 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.516041 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.519087 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:03.519093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:03.519099 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:03.519156 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:03.522670 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:03.523474 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:03.523479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:03.523483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:03.523670 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:03.524227 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:03.524232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:03.524235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:03.524599 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:03.524807 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:03.525276 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 11:33:03.525583 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":421,"thread_start_us":289,"threads_started":1}
I20260807 11:33:03.526865 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.499954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:48845 (local address 127.24.103.126:42305)
0807 11:33:03.500222 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:03.500232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:03.501835 (+ 1603us) server_negotiation.cc:407] Connection header received
0807 11:33:03.505004 (+ 3169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:03.505015 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:03.505139 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:03.505236 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:03.508156 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.509233 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.511883 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.512461 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.512623 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.517705 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:03.517711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:03.517716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:03.517754 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:03.522534 (+ 4780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:03.523658 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:03.523662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:03.523664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:03.523867 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:03.524468 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:03.524472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:03.524474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:03.524866 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:03.525040 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:03.525982 (+ 942us) server_negotiation.cc:299] Negotiation successful
0807 11:33:03.526293 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":86}
I20260807 11:33:03.528266 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.500644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:48845)
0807 11:33:03.501578 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:03.501636 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:03.504568 (+ 2932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:03.505944 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:03.505961 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:03.506924 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:33:03.507938 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.507967 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.509452 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.509456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.511708 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.511723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.512713 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.512723 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.513011 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.513858 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:33:03.513892 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:33:03.517444 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:33:03.522715 (+ 5271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:03.522728 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:03.522749 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:03.523510 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:03.524002 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:03.524006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:03.524008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:03.524299 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:03.525018 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:33:03.525024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:33:03.525493 (+ 469us) client_negotiation.cc:769] Sending connection context
0807 11:33:03.525858 (+ 365us) client_negotiation.cc:240] Negotiation successful
0807 11:33:03.526540 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":343,"threads_started":1}
I20260807 11:33:03.529579 9552 heartbeater.cc:347] Connected to a master server at 127.24.103.126:42305
I20260807 11:33:03.529909 9552 heartbeater.cc:464] Registering TS with master...
I20260807 11:33:03.530646 9552 heartbeater.cc:511] Master 127.24.103.126:42305 requested a full tablet report, sending...
I20260807 11:33:03.532253 9013 ts_manager.cc:194] Registered new tserver with Master: 79850c65db02473cb81f06f29889ae9e (127.24.103.67:37847)
I20260807 11:33:03.533136 9013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:48845
I20260807 11:33:03.541435 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:33:03.548801 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.542433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.24.103.126:42305)
0807 11:33:03.542620 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:03.542625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:03.543144 (+ 519us) server_negotiation.cc:407] Connection header received
0807 11:33:03.543309 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:03.543313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:03.543436 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:03.543557 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:33:03.543977 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.544832 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.546387 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.546821 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.546922 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.547493 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:33:03.547973 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:33:03.548108 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:03.548502 (+ 394us) server_negotiation.cc:299] Negotiation successful
0807 11:33:03.548580 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 11:33:03.552564 9229 heartbeater.cc:503] Master 127.24.103.126:42305 was elected leader, sending a full tablet report...
I20260807 11:33:03.549867 9013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:33:03.558180 9013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:33:03.571869 9155 tablet_service.cc:1467] Processing CreateTablet for tablet 690bf8d6d677487e86761b28b8e4fb0a (DEFAULT_TABLE table=test-table [id=bb14d4665c0d43c6ab00afe3061ad50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:03.572968 9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690bf8d6d677487e86761b28b8e4fb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:03.581424 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.570970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.24.103.66:36899)
0807 11:33:03.573875 (+ 2905us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:03.573886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:03.573913 (+ 27us) server_negotiation.cc:407] Connection header received
0807 11:33:03.574018 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:03.574026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:03.574252 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:03.574443 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:03.575005 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.576551 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.579046 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.580191 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.580353 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.580561 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:03.580922 (+ 361us) server_negotiation.cc:299] Negotiation successful
0807 11:33:03.581079 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2737,"thread_start_us":2565,"threads_started":1}
I20260807 11:33:03.581513 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.570411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:36899 (local address 127.0.0.1:42566)
0807 11:33:03.571769 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:03.571787 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:03.571897 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:03.574457 (+ 2560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:03.574462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:03.574491 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:03.574816 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.574829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.576707 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.576713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.578838 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.578853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.580403 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.580410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.580713 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.580740 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 11:33:03.580850 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 11:33:03.580992 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"mutex_wait_us":102,"thread_start_us":1032,"threads_started":1}
I20260807 11:33:03.582372 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.569917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37847 (local address 127.0.0.1:52564)
0807 11:33:03.570859 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:03.570893 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:03.571093 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:03.573109 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:03.573114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:03.573140 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:03.573507 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.573524 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.576175 (+ 2651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.576182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.578324 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.578341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.580573 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.580583 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.580871 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.580926 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 11:33:03.581008 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 11:33:03.581217 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":578,"threads_started":1}
I20260807 11:33:03.583947 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.570269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.24.103.67:37847)
0807 11:33:03.572510 (+ 2241us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:03.572521 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:03.572557 (+ 36us) server_negotiation.cc:407] Connection header received
0807 11:33:03.572662 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:03.572670 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:03.572908 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:03.573109 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:03.573764 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.576018 (+ 2254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.578508 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.580261 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.582814 (+ 2553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.583159 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:03.583260 (+ 101us) server_negotiation.cc:299] Negotiation successful
0807 11:33:03.583488 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2016,"thread_start_us":2296,"threads_started":1}
I20260807 11:33:03.585621 9317 tablet_service.cc:1467] Processing CreateTablet for tablet 690bf8d6d677487e86761b28b8e4fb0a (DEFAULT_TABLE table=test-table [id=bb14d4665c0d43c6ab00afe3061ad50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:03.586694 9569 tablet_bootstrap.cc:492] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6: Bootstrap starting.
I20260807 11:33:03.587147 9317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690bf8d6d677487e86761b28b8e4fb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:03.589184 9569 tablet_bootstrap.cc:654] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:03.590046 9477 tablet_service.cc:1467] Processing CreateTablet for tablet 690bf8d6d677487e86761b28b8e4fb0a (DEFAULT_TABLE table=test-table [id=bb14d4665c0d43c6ab00afe3061ad50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:03.591315 9569 tablet_bootstrap.cc:492] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6: No bootstrap required, opened a new log
I20260807 11:33:03.591491 9569 ts_tablet_manager.cc:1397] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 11:33:03.591518 9477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690bf8d6d677487e86761b28b8e4fb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:03.592191 9569 raft_consensus.cc:348] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:03.592465 9569 raft_consensus.cc:374] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:03.592573 9569 raft_consensus.cc:729] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6182262691442daa97990f9959c3a6, State: Initialized, Role: FOLLOWER
I20260807 11:33:03.592859 9569 consensus_queue.cc:260] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:03.593747 9569 ts_tablet_manager.cc:1428] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 11:33:03.603379 9573 tablet_bootstrap.cc:492] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f: Bootstrap starting.
I20260807 11:33:03.606313 9573 tablet_bootstrap.cc:654] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:03.606969 9574 tablet_bootstrap.cc:492] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e: Bootstrap starting.
I20260807 11:33:03.607544 9573 log.cc:824] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:03.610496 9574 tablet_bootstrap.cc:654] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:03.610960 9573 tablet_bootstrap.cc:492] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f: No bootstrap required, opened a new log
I20260807 11:33:03.611308 9573 ts_tablet_manager.cc:1397] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20260807 11:33:03.611757 9574 log.cc:824] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:03.614594 9574 tablet_bootstrap.cc:492] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e: No bootstrap required, opened a new log
I20260807 11:33:03.614866 9574 ts_tablet_manager.cc:1397] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260807 11:33:03.620350 9573 raft_consensus.cc:348] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:03.620760 9573 raft_consensus.cc:374] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:03.620882 9573 raft_consensus.cc:729] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128a484d329d43728bd5447f6af1763f, State: Initialized, Role: FOLLOWER
I20260807 11:33:03.621582 9573 consensus_queue.cc:260] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:03.622342 9574 raft_consensus.cc:348] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:03.622826 9398 heartbeater.cc:503] Master 127.24.103.126:42305 was elected leader, sending a full tablet report...
I20260807 11:33:03.623396 9573 ts_tablet_manager.cc:1428] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260807 11:33:03.623621 9574 raft_consensus.cc:374] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:03.623736 9574 raft_consensus.cc:729] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79850c65db02473cb81f06f29889ae9e, State: Initialized, Role: FOLLOWER
I20260807 11:33:03.624323 9574 consensus_queue.cc:260] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:03.625538 9552 heartbeater.cc:503] Master 127.24.103.126:42305 was elected leader, sending a full tablet report...
I20260807 11:33:03.625985 9574 ts_tablet_manager.cc:1428] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260807 11:33:03.664522 9577 raft_consensus.cc:482] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:33:03.664873 9577 raft_consensus.cc:504] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:03.666684 9577 leader_election.cc:288] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79850c65db02473cb81f06f29889ae9e (127.24.103.67:37847), ee6182262691442daa97990f9959c3a6 (127.24.103.65:45641)
I20260807 11:33:03.674038 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.667273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:54047 (local address 127.24.103.67:37847)
0807 11:33:03.667539 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:03.667546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:03.667935 (+ 389us) server_negotiation.cc:407] Connection header received
0807 11:33:03.668225 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:03.668233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:03.668329 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:03.668433 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:03.669150 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.669907 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.672177 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.673114 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.673302 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.673379 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:03.673646 (+ 267us) server_negotiation.cc:299] Negotiation successful
0807 11:33:03.673779 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:33:03.674186 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.667113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:37847 (local address 127.24.103.66:54047)
0807 11:33:03.667814 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:03.667832 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:03.668023 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:03.668573 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:03.668581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:03.668626 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:03.668999 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.669008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.670039 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.670047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.672008 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.672031 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.673328 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.673334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.673485 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.673509 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 11:33:03.673620 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 11:33:03.673757 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":385,"threads_started":1}
I20260807 11:33:03.674783 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.667965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:45641 (local address 127.24.103.66:36167)
0807 11:33:03.668657 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:03.668678 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:03.668823 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:03.669301 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:03.669306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:03.669341 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:03.669690 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.669699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.671040 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.671049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.672601 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:03.672613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.673960 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.673966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:03.674200 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.674238 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 11:33:03.674326 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 11:33:03.674466 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":322,"threads_started":1}
I20260807 11:33:03.674846 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.668105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:36167 (local address 127.24.103.65:45641)
0807 11:33:03.668990 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:03.668994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:03.669009 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:33:03.669067 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:03.669071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:03.669127 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:03.669287 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:03.669881 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.670864 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.672767 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.673728 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.673929 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.674000 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:03.674368 (+ 368us) server_negotiation.cc:299] Negotiation successful
0807 11:33:03.674526 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":594,"threads_started":1}
I20260807 11:33:03.675356 9501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690bf8d6d677487e86761b28b8e4fb0a" candidate_uuid: "128a484d329d43728bd5447f6af1763f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79850c65db02473cb81f06f29889ae9e" is_pre_election: true
I20260807 11:33:03.675695 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690bf8d6d677487e86761b28b8e4fb0a" candidate_uuid: "128a484d329d43728bd5447f6af1763f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6182262691442daa97990f9959c3a6" is_pre_election: true
I20260807 11:33:03.675880 9501 raft_consensus.cc:2454] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 128a484d329d43728bd5447f6af1763f in term 0.
I20260807 11:33:03.676196 9177 raft_consensus.cc:2454] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 128a484d329d43728bd5447f6af1763f in term 0.
I20260807 11:33:03.676632 9259 leader_election.cc:302] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128a484d329d43728bd5447f6af1763f, 79850c65db02473cb81f06f29889ae9e; no voters:
I20260807 11:33:03.677024 9577 raft_consensus.cc:2781] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:33:03.677211 9577 raft_consensus.cc:482] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:33:03.677338 9577 raft_consensus.cc:3037] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:03.679664 9577 raft_consensus.cc:504] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:03.680311 9577 leader_election.cc:288] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [CANDIDATE]: Term 1 election: Requested vote from peers 79850c65db02473cb81f06f29889ae9e (127.24.103.67:37847), ee6182262691442daa97990f9959c3a6 (127.24.103.65:45641)
I20260807 11:33:03.680783 9501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690bf8d6d677487e86761b28b8e4fb0a" candidate_uuid: "128a484d329d43728bd5447f6af1763f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79850c65db02473cb81f06f29889ae9e"
I20260807 11:33:03.680873 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690bf8d6d677487e86761b28b8e4fb0a" candidate_uuid: "128a484d329d43728bd5447f6af1763f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6182262691442daa97990f9959c3a6"
I20260807 11:33:03.681002 9501 raft_consensus.cc:3037] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:03.681193 9177 raft_consensus.cc:3037] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:03.683442 9501 raft_consensus.cc:2454] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 128a484d329d43728bd5447f6af1763f in term 1.
I20260807 11:33:03.683705 9177 raft_consensus.cc:2454] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 128a484d329d43728bd5447f6af1763f in term 1.
I20260807 11:33:03.683984 9259 leader_election.cc:302] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128a484d329d43728bd5447f6af1763f, 79850c65db02473cb81f06f29889ae9e; no voters:
I20260807 11:33:03.684348 9577 raft_consensus.cc:2781] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:33:03.684867 9577 raft_consensus.cc:686] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 1 LEADER]: Becoming Leader. State: Replica: 128a484d329d43728bd5447f6af1763f, State: Running, Role: LEADER
I20260807 11:33:03.685431 9577 consensus_queue.cc:237] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } } peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } }
I20260807 11:33:03.691085 9013 catalog_manager.cc:5495] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f reported cstate change: term changed from 0 to 1, leader changed from <none> to 128a484d329d43728bd5447f6af1763f (127.24.103.66). New cstate: current_term: 1 leader_uuid: "128a484d329d43728bd5447f6af1763f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "128a484d329d43728bd5447f6af1763f" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 36899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 } health_report { overall_health: UNKNOWN } } }
I20260807 11:33:03.708640 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.702858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.24.103.66:36899)
0807 11:33:03.703104 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:03.703112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:03.703136 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:33:03.703421 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:03.703428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:03.703545 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:03.703732 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:33:03.704198 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.704928 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.706435 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.706856 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.706988 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.707542 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:33:03.708039 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:33:03.708210 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:03.708270 (+ 60us) server_negotiation.cc:299] Negotiation successful
0807 11:33:03.708352 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 11:33:03.717123 9501 raft_consensus.cc:1261] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e [term 1 FOLLOWER]: Refusing update from remote peer 128a484d329d43728bd5447f6af1763f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:33:03.717186 9177 raft_consensus.cc:1261] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Refusing update from remote peer 128a484d329d43728bd5447f6af1763f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:33:03.718039 9577 consensus_queue.cc:1035] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6182262691442daa97990f9959c3a6" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 45641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:33:03.718379 9583 consensus_queue.cc:1035] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [LEADER]: Connected to new peer: Peer: permanent_uuid: "79850c65db02473cb81f06f29889ae9e" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:33:03.742312 9013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:33:03.742617 9013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:33:03.746212 9013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Sending DeleteTablet for 3 replicas of tablet 690bf8d6d677487e86761b28b8e4fb0a
I20260807 11:33:03.747341 9317 tablet_service.cc:1514] Processing DeleteTablet for tablet 690bf8d6d677487e86761b28b8e4fb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:03 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:42566
I20260807 11:33:03.747365 9155 tablet_service.cc:1514] Processing DeleteTablet for tablet 690bf8d6d677487e86761b28b8e4fb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:03 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:39978
I20260807 11:33:03.747392 9477 tablet_service.cc:1514] Processing DeleteTablet for tablet 690bf8d6d677487e86761b28b8e4fb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:03 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:52564
I20260807 11:33:03.748230 9597 tablet_replica.cc:330] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6: stopping tablet replica
I20260807 11:33:03.748577 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 9057
I20260807 11:33:03.748555 9597 raft_consensus.cc:2229] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:33:03.749068 9597 raft_consensus.cc:2258] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:03.752346 9599 tablet_replica.cc:330] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f: stopping tablet replica
I20260807 11:33:03.752876 9599 raft_consensus.cc:2229] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:33:03.753113 9597 ts_tablet_manager.cc:1905] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:03.753703 9599 raft_consensus.cc:2258] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:03.762571 9597 ts_tablet_manager.cc:1918] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:03.762773 9597 log.cc:1196] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/690bf8d6d677487e86761b28b8e4fb0a
I20260807 11:33:03.764776 9598 tablet_replica.cc:330] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e: stopping tablet replica
I20260807 11:33:03.765322 9599 ts_tablet_manager.cc:1905] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:03.765396 9598 raft_consensus.cc:2229] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:33:03.766055 9598 raft_consensus.cc:2258] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:03.766758 9597 ts_tablet_manager.cc:1939] T 690bf8d6d677487e86761b28b8e4fb0a P ee6182262691442daa97990f9959c3a6: Deleting consensus metadata
I20260807 11:33:03.768046 8996 catalog_manager.cc:4841] TS ee6182262691442daa97990f9959c3a6 (127.24.103.65:45641): tablet 690bf8d6d677487e86761b28b8e4fb0a (table test-table [id=bb14d4665c0d43c6ab00afe3061ad50f]) successfully deleted
I20260807 11:33:03.769644 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.750646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.24.103.65:45641)
0807 11:33:03.750878 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:03.750887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:03.750912 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:33:03.750985 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:03.750994 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:03.751136 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:03.751267 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:03.753189 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.754328 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.756378 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.757056 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.757206 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.760964 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:03.760983 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:03.761004 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:03.761057 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:03.765326 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:03.766309 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:03.766319 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:03.766322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:03.766493 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:03.766975 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:03.766981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:03.766984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:03.767512 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:03.767844 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:03.768889 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0807 11:33:03.769243 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:33:03.770345 9224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:03.773187 9598 ts_tablet_manager.cc:1905] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:03.775098 9599 ts_tablet_manager.cc:1918] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:03.775287 9599 log.cc:1196] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/690bf8d6d677487e86761b28b8e4fb0a
I20260807 11:33:03.775830 9599 ts_tablet_manager.cc:1939] T 690bf8d6d677487e86761b28b8e4fb0a P 128a484d329d43728bd5447f6af1763f: Deleting consensus metadata
I20260807 11:33:03.777392 8994 catalog_manager.cc:4841] TS 128a484d329d43728bd5447f6af1763f (127.24.103.66:36899): tablet 690bf8d6d677487e86761b28b8e4fb0a (table test-table [id=bb14d4665c0d43c6ab00afe3061ad50f]) successfully deleted
I20260807 11:33:03.781198 9598 ts_tablet_manager.cc:1918] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:03.781427 9598 log.cc:1196] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/690bf8d6d677487e86761b28b8e4fb0a
I20260807 11:33:03.782148 9598 ts_tablet_manager.cc:1939] T 690bf8d6d677487e86761b28b8e4fb0a P 79850c65db02473cb81f06f29889ae9e: Deleting consensus metadata
I20260807 11:33:03.784471 8996 catalog_manager.cc:4841] TS 79850c65db02473cb81f06f29889ae9e (127.24.103.67:37847): tablet 690bf8d6d677487e86761b28b8e4fb0a (table test-table [id=bb14d4665c0d43c6ab00afe3061ad50f]) successfully deleted
I20260807 11:33:03.869478 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 9057
I20260807 11:33:03.900715 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 9233
I20260807 11:33:03.918802 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:03.901894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.24.103.66:36899)
0807 11:33:03.902088 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:03.902093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:03.902113 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:33:03.902304 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:03.902308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:03.902393 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:03.902486 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:03.904659 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.905585 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.907746 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:03.908234 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:03.908350 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:03.911986 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:03.912002 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:03.912032 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:03.912083 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:03.915442 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:03.916389 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:03.916397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:03.916400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:03.916590 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:03.917035 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:03.917039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:03.917041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:03.917508 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:03.917823 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:03.918190 (+ 367us) server_negotiation.cc:299] Negotiation successful
0807 11:33:03.918408 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 11:33:03.919656 9391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:04.036263 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 9233
I20260807 11:33:04.059612 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 9401
I20260807 11:33:04.074234 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:04.060523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.24.103.67:37847)
0807 11:33:04.060718 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:04.060723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:04.060742 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:33:04.060877 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:04.060881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:04.060952 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:04.061038 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:04.062484 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:04.063337 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:04.064660 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:04.065047 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:04.065154 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:04.068341 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:04.068355 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:04.068371 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:04.068410 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:04.071279 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:04.072060 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:04.072068 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:04.072070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:04.072219 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:04.072618 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:04.072623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:04.072626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:04.073028 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:04.073332 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:04.073659 (+ 327us) server_negotiation.cc:299] Negotiation successful
0807 11:33:04.073893 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:33:04.074903 9546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:04.165840 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 9401
I20260807 11:33:04.187935 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 8975
I20260807 11:33:04.189309 9047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:04.306108 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 8975
I20260807 11:33:04.335987 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42305
--webserver_interface=127.24.103.126
--webserver_port=34591
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:42305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:33:04.686448 9607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:04.686777 9607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:04.686946 9607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:04.694710 9607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:33:04.694836 9607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:04.694940 9607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:04.695096 9607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:33:04.695154 9607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:33:04.705555 9607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:42305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:42305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=34591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:04.707120 9607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:04.708472 9607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:04.720597 9614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:04.720660 9615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:04.720681 9618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:04.721640 9607 server_base.cc:1034] running on GCE node
I20260807 11:33:04.722478 9607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:04.724069 9607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:04.725301 9607 hybrid_clock.cc:648] HybridClock initialized: now 1786102384725241 us; error 41 us; skew 500 ppm
I20260807 11:33:04.725678 9607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:04 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102384, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:04.729126 9607 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:33:04.730697 9607 webserver.cc:460] Webserver started at http://127.24.103.126:34591/ using document root <none> and password file <none>
I20260807 11:33:04.731462 9607 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:04.731608 9607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:04.737263 9607 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:33:04.740324 9627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:04.741734 9607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:33:04.741927 9607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "a5f9fa9bf2fc43108bf298e006972b3e"
format_stamp: "Formatted at 2026-08-07 11:32:58 on dist-test-slave-wl2h"
server_key: "9bccf25cbbc856a944056346a5ce1bae"
server_key_iv: "6fcdcbcca7679173256351b218f13094"
server_key_version: "encryptionkey@0"
I20260807 11:33:04.743042 9607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:04.756467 9607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:04.760953 9607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:04.761248 9607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:04.781190 9607 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:42305
I20260807 11:33:04.781283 9683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:42305 every 8 connection(s)
I20260807 11:33:04.782840 9607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:33:04.785640 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 9607
I20260807 11:33:04.786767 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:45641
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:42305
--webserver_port=41009
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:33:04.793217 9684 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:33:04.798569 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Bootstrap starting.
I20260807 11:33:04.820057 9684 log.cc:824] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:04.836755 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:33:04.837337 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e: Bootstrap complete.
I20260807 11:33:04.846920 9684 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:33:04.847950 9684 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f9fa9bf2fc43108bf298e006972b3e, State: Initialized, Role: FOLLOWER
I20260807 11:33:04.848551 9684 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:33:04.848799 9684 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:33:04.848937 9684 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:33:04.849123 9684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:33:04.853202 9684 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:33:04.853716 9684 leader_election.cc:302] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5f9fa9bf2fc43108bf298e006972b3e; no voters:
I20260807 11:33:04.854429 9684 leader_election.cc:288] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 11:33:04.854575 9688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:33:04.855821 9688 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [term 3 LEADER]: Becoming Leader. State: Replica: a5f9fa9bf2fc43108bf298e006972b3e, State: Running, Role: LEADER
I20260807 11:33:04.856411 9688 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } }
I20260807 11:33:04.856761 9684 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:33:04.859265 9690 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5f9fa9bf2fc43108bf298e006972b3e. Latest consensus state: current_term: 3 leader_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } } }
I20260807 11:33:04.859298 9689 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9fa9bf2fc43108bf298e006972b3e" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 42305 } } }
I20260807 11:33:04.859661 9689 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: This master's current role is: LEADER
I20260807 11:33:04.859666 9690 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f9fa9bf2fc43108bf298e006972b3e [sys.catalog]: This master's current role is: LEADER
I20260807 11:33:04.872170 9695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:33:04.880916 9695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:33:04.882491 9695 catalog_manager.cc:1259] Loaded cluster ID: 8dddcdf96f8048cd859f158387a4f592
I20260807 11:33:04.882615 9695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 11:33:04.884886 9695 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f65d60dd9ad google::LogMessage::Fail() at ??:0
@ 0x7f65d60e088a google::LogMessage::SendToLog() at ??:0
@ 0x7f65d60dd4de google::LogMessage::Flush() at ??:0
@ 0x7f65d60dea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f65ebb10512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f65ebb0d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f65ebb95610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f65ebb95439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f65d8d29345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f65d8d2c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f65d8d2c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f65d8d07eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f65dbfe16db start_thread at ??:0
@ 0x7f65d4f7761f clone at ??:0
W20260807 11:33:05.223879 9686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:05.224177 9686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:05.224350 9686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:05.231930 9686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:05.232079 9686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:05.232316 9686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:33:05.242408 9686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:45641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=41009
--tserver_master_addrs=127.24.103.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:05.243850 9686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:05.245296 9686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:05.259407 9729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:05.259339 9730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:05.259536 9732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:05.260182 9686 server_base.cc:1034] running on GCE node
I20260807 11:33:05.260857 9686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:05.261895 9686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:05.263127 9686 hybrid_clock.cc:648] HybridClock initialized: now 1786102385263077 us; error 55 us; skew 500 ppm
I20260807 11:33:05.263464 9686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:05 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102385, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:05.266677 9686 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:33:05.268240 9686 webserver.cc:460] Webserver started at http://127.24.103.65:41009/ using document root <none> and password file <none>
I20260807 11:33:05.268965 9686 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:05.269135 9686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:05.274842 9686 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:33:05.277689 9741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:05.279294 9686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:33:05.279496 9686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "ee6182262691442daa97990f9959c3a6"
format_stamp: "Formatted at 2026-08-07 11:32:59 on dist-test-slave-wl2h"
server_key: "5698e7d737a4ce2a7a048da42932da3b"
server_key_iv: "0c31a7a212a97a7f6c41fd9c61d5d7c7"
server_key_version: "encryptionkey@0"
I20260807 11:33:05.280705 9686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:05.317617 9686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:05.322788 9686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:05.323135 9686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:05.324174 9686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:05.325587 9686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:05.325678 9686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:05.325754 9686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:05.325829 9686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:05.362874 9686 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:45641
I20260807 11:33:05.362995 9867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:45641 every 8 connection(s)
I20260807 11:33:05.364529 9686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:33:05.369966 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:05.368019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:52037)
0807 11:33:05.368829 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:05.369085 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":323,"threads_started":1}
W20260807 11:33:05.370692 9868 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:42305 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:42305: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
I20260807 11:33:05.371703 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:05.371306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:44017)
0807 11:33:05.371476 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:05.371546 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 11:33:05.372368 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 9686
I20260807 11:33:05.372879 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:05.372482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:55887)
0807 11:33:05.372634 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:05.372731 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260807 11:33:05.373117 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:36899
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:42305
--webserver_port=38523
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:33:05.373189 9868 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 11:33:05.724218 9872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:05.724530 9872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:05.724628 9872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:05.732095 9872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:05.732220 9872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:05.732347 9872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:33:05.742241 9872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:36899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=38523
--tserver_master_addrs=127.24.103.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:05.743628 9872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:05.744915 9872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:05.758505 9877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:05.758507 9878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:05.758553 9881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:05.760021 9872 server_base.cc:1034] running on GCE node
I20260807 11:33:05.760957 9872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:05.762072 9872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:05.763310 9872 hybrid_clock.cc:648] HybridClock initialized: now 1786102385763257 us; error 51 us; skew 500 ppm
I20260807 11:33:05.763644 9872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:05 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102385, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:05.766979 9872 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:33:05.768574 9872 webserver.cc:460] Webserver started at http://127.24.103.66:38523/ using document root <none> and password file <none>
I20260807 11:33:05.769304 9872 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:05.769456 9872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:05.775128 9872 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:33:05.778023 9889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:05.779788 9872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:33:05.779996 9872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "128a484d329d43728bd5447f6af1763f"
format_stamp: "Formatted at 2026-08-07 11:32:59 on dist-test-slave-wl2h"
server_key: "d4caa51690d8aa40ca2d5bccf79ccd82"
server_key_iv: "a0ae35e553bd1a4571c2edf1b44e2428"
server_key_version: "encryptionkey@0"
I20260807 11:33:05.781100 9872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:05.798955 9872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:05.803702 9872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:05.804021 9872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:05.805148 9872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:05.806594 9872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:05.806741 9872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:05.806882 9872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:05.806978 9872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:05.843596 9872 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:36899
I20260807 11:33:05.843693 10026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:36899 every 8 connection(s)
I20260807 11:33:05.845145 9872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:33:05.846863 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 9872
I20260807 11:33:05.847615 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:37847
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:42305
--webserver_port=36477
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:33:05.850584 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:05.848307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:40411)
0807 11:33:05.849186 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:05.849454 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":325,"threads_started":1}
W20260807 11:33:05.851279 10027 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:42305 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:42305: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
I20260807 11:33:05.852262 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:05.851813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:52825)
0807 11:33:05.852021 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:05.852118 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 11:33:05.853261 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:05.852835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:51395)
0807 11:33:05.853004 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:05.853105 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 11:33:05.853577 10027 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 11:33:06.195113 10031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:06.195418 10031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:06.195585 10031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:06.203081 10031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:06.203188 10031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:06.203395 10031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:33:06.213778 10031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:37847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=36477
--tserver_master_addrs=127.24.103.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:06.215286 10031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:06.217231 10031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:06.230788 10036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:06.230860 10037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:06.230883 10039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:06.236670 10031 server_base.cc:1034] running on GCE node
I20260807 11:33:06.237398 10031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:06.238669 10031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:06.239923 10031 hybrid_clock.cc:648] HybridClock initialized: now 1786102386239882 us; error 45 us; skew 500 ppm
I20260807 11:33:06.240348 10031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:06 dist-test-slave-wl2h krb5kdc[8342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102386, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:06.243757 10031 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:33:06.245479 10031 webserver.cc:460] Webserver started at http://127.24.103.67:36477/ using document root <none> and password file <none>
I20260807 11:33:06.246213 10031 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:06.246376 10031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:06.251835 10031 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:33:06.254474 10047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:06.255779 10031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:33:06.255951 10031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "79850c65db02473cb81f06f29889ae9e"
format_stamp: "Formatted at 2026-08-07 11:33:00 on dist-test-slave-wl2h"
server_key: "41a6a4b4e95077da1425f224168da4d0"
server_key_iv: "9c3b2d649255b18efc7493677970a2dd"
server_key_version: "encryptionkey@0"
I20260807 11:33:06.256901 10031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:06.293114 10031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:06.297755 10031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:06.298050 10031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:06.299167 10031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:06.300581 10031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:06.300707 10031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:06.300854 10031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:06.300945 10031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:06.340163 10031 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:37847
I20260807 11:33:06.340237 10172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:37847 every 8 connection(s)
I20260807 11:33:06.341727 10031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:33:06.346750 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:06.344858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:57839)
0807 11:33:06.345651 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:06.345886 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":326,"threads_started":1}
W20260807 11:33:06.347479 10173 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:42305 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:42305: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
I20260807 11:33:06.348328 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:06.347988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:50829)
0807 11:33:06.348143 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:06.348227 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 11:33:06.349258 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:06.348952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:52591)
0807 11:33:06.349072 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:06.349150 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 11:33:06.349645 10173 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 11:33:06.350775 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 10031
I20260807 11:33:06.374744 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:06.373781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:42331)
0807 11:33:06.374302 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:06.374423 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":236,"threads_started":1}
I20260807 11:33:06.855173 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:06.854261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:42165)
0807 11:33:06.854745 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:06.854876 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":252,"threads_started":1}
I20260807 11:33:07.351150 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:07.350230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:53823)
0807 11:33:07.350712 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:07.350842 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":225,"threads_started":1}
I20260807 11:33:07.377022 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:07.375821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:42017)
0807 11:33:07.376772 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:07.376856 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":324,"threads_started":1}
I20260807 11:33:07.856940 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:07.856200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:56941)
0807 11:33:07.856673 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:07.856762 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":229,"threads_started":1}
I20260807 11:33:08.352903 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:08.352109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:56397)
0807 11:33:08.352628 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:08.352719 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":277,"threads_started":1}
I20260807 11:33:08.379186 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:08.378060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:42975)
0807 11:33:08.378732 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:08.378850 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":289,"threads_started":1}
I20260807 11:33:08.858788 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:08.858025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:57135)
0807 11:33:08.858487 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:08.858604 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":242,"threads_started":1}
I20260807 11:33:09.354682 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:09.353922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:43675)
0807 11:33:09.354412 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:09.354499 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
I20260807 11:33:09.381507 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:09.380258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:56895)
0807 11:33:09.381099 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:09.381231 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":367,"threads_started":1}
I20260807 11:33:09.860903 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:09.859953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:39827)
0807 11:33:09.860578 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:09.860670 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":347,"threads_started":1}
I20260807 11:33:10.356619 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:10.355806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:45875)
0807 11:33:10.356341 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:10.356432 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":276,"threads_started":1}
I20260807 11:33:10.383553 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:10.382651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:53381)
0807 11:33:10.383273 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:10.383365 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":269,"threads_started":1}
I20260807 11:33:10.862815 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:10.862017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:36257)
0807 11:33:10.862526 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:10.862626 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":265,"threads_started":1}
I20260807 11:33:11.358443 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:11.357629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:46349)
0807 11:33:11.358174 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:11.358257 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":261,"threads_started":1}
I20260807 11:33:11.385648 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:11.384668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:47259)
0807 11:33:11.385347 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:11.385466 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":308,"threads_started":1}
I20260807 11:33:11.864601 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:11.863796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:38625)
0807 11:33:11.864313 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:11.864410 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":275,"threads_started":1}
I20260807 11:33:12.360124 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:12.359337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:34685)
0807 11:33:12.359867 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:12.359960 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":290,"threads_started":1}
I20260807 11:33:12.387459 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:12.386553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:34393)
0807 11:33:12.387151 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:12.387245 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":229,"threads_started":1}
I20260807 11:33:12.866570 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:12.865704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:56405)
0807 11:33:12.866240 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:12.866328 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":213,"threads_started":1}
I20260807 11:33:13.361896 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:13.361048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:50593)
0807 11:33:13.361595 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:13.361682 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":222,"threads_started":1}
I20260807 11:33:13.389138 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:13.388349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:59199)
0807 11:33:13.388873 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:13.388963 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":263,"threads_started":1}
I20260807 11:33:13.868379 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:13.867632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:49019)
0807 11:33:13.868111 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:13.868204 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":238,"threads_started":1}
I20260807 11:33:14.363554 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:14.362771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:46173)
0807 11:33:14.363314 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:14.363396 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":311,"threads_started":1}
I20260807 11:33:14.390825 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:14.390015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:53705)
0807 11:33:14.390509 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:14.390634 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":244,"threads_started":1}
I20260807 11:33:14.870229 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:14.869403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:34215)
0807 11:33:14.869938 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:14.870030 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":268,"threads_started":1}
I20260807 11:33:15.365657 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:15.364449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:55201)
0807 11:33:15.365380 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:15.365486 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":609,"threads_started":1}
I20260807 11:33:15.392582 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:15.391810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:46337)
0807 11:33:15.392295 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:15.392394 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":268,"threads_started":1}
I20260807 11:33:15.872287 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:15.871334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:35951)
0807 11:33:15.871934 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:15.872116 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":284,"threads_started":1}
I20260807 11:33:16.367306 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:16.366544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:38105)
0807 11:33:16.366998 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:16.367141 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":253,"threads_started":1}
I20260807 11:33:16.394248 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:16.393494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:44857)
0807 11:33:16.393939 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:16.394026 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":224,"threads_started":1}
I20260807 11:33:16.873975 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:16.873243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:43419)
0807 11:33:16.873681 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:16.873789 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":205,"threads_started":1}
I20260807 11:33:17.368899 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:17.368184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:35549)
0807 11:33:17.368632 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:17.368725 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":226,"threads_started":1}
I20260807 11:33:17.396147 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:17.395338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:57749)
0807 11:33:17.395826 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:17.395917 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260807 11:33:17.875806 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:17.875082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:33827)
0807 11:33:17.875559 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:17.875645 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":248,"threads_started":1}
I20260807 11:33:18.370477 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:18.369831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:42463)
0807 11:33:18.370250 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:18.370331 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":213,"threads_started":1}
I20260807 11:33:18.398123 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:18.397239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:33201)
0807 11:33:18.397784 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:18.397882 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":267,"threads_started":1}
I20260807 11:33:18.877651 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:18.876852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:41143)
0807 11:33:18.877362 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:18.877452 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":268,"threads_started":1}
I20260807 11:33:19.372437 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:19.371480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:58555)
0807 11:33:19.372079 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:19.372185 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":315,"threads_started":1}
I20260807 11:33:19.400012 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:19.399176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:51753)
0807 11:33:19.399712 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:19.399809 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":274,"threads_started":1}
I20260807 11:33:19.879607 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:19.878687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:41825)
0807 11:33:19.879307 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:19.879404 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":274,"threads_started":1}
I20260807 11:33:20.374413 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:20.373408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:43041)
0807 11:33:20.373987 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:20.374112 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":315,"threads_started":1}
I20260807 11:33:20.402057 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:20.401222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.65:54711)
0807 11:33:20.401757 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:20.401857 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":298,"threads_started":1}
I20260807 11:33:20.881541 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:20.880790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.66:47723)
0807 11:33:20.881253 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:20.881351 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260807 11:33:21.363144 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 9686
I20260807 11:33:21.376628 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:21.375676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:42305 (local address 127.24.103.67:52719)
0807 11:33:21.376171 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:21.376313 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:42305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":214,"threads_started":1}
I20260807 11:33:21.390882 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:21.364386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.24.103.65:45641)
0807 11:33:21.365591 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:21.365618 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:21.365676 (+ 58us) server_negotiation.cc:407] Connection header received
0807 11:33:21.367870 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:21.367880 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:21.368305 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:21.368602 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:21.370109 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:21.371598 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:21.373158 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:21.373802 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:21.373941 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:21.384392 (+ 10451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:21.384401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:21.384418 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:21.384459 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:21.387648 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:21.388434 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:21.388447 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:21.388457 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:21.388690 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:21.389073 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:21.389076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:21.389078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:21.389619 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:21.389863 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:21.390266 (+ 403us) server_negotiation.cc:299] Negotiation successful
0807 11:33:21.390603 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":831,"threads_started":1}
I20260807 11:33:21.392340 9863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:21.480115 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 9686
I20260807 11:33:21.502462 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 9872
I20260807 11:33:21.521618 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:21.503586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.24.103.66:36899)
0807 11:33:21.504126 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:21.504141 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:21.504181 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:33:21.505830 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:21.505840 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:21.506275 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:21.506659 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:21.508310 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:21.509570 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:21.510903 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:21.511657 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:21.511877 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:21.514665 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:21.514674 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:21.514697 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:21.514742 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:21.517780 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:21.518528 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:21.518541 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:21.518548 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:21.518731 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:21.519432 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:21.519439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:21.519443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:21.519987 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:21.520322 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:21.520849 (+ 527us) server_negotiation.cc:299] Negotiation successful
0807 11:33:21.521311 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":262,"threads_started":1}
I20260807 11:33:21.523285 10021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:21.610266 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 9872
I20260807 11:33:21.632752 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 10031
I20260807 11:33:21.652733 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:21.634054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.24.103.67:37847)
0807 11:33:21.634770 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:21.634783 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:21.634841 (+ 58us) server_negotiation.cc:407] Connection header received
0807 11:33:21.636694 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:21.636711 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:21.637157 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:21.637468 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:21.638921 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:21.640272 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:21.641697 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:21.642369 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:21.642601 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:21.646088 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:21.646096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:21.646113 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:21.646153 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:21.649202 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:21.649949 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:21.649956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:21.649963 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:21.650174 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:21.650553 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:21.650557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:21.650559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:21.651237 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:21.651472 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:21.651937 (+ 465us) server_negotiation.cc:299] Negotiation successful
0807 11:33:21.652420 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":392,"threads_started":1}
I20260807 11:33:21.654352 10168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:21.741312 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 10031
2026-08-07T11:33:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25287 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:33:21 dist-test-slave-wl2h krb5kdc[10239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:33:21 dist-test-slave-wl2h krb5kdc[10239](info): set up 2 sockets
Aug 07 11:33:21 dist-test-slave-wl2h krb5kdc[10239](info): commencing operation
krb5kdc: starting...
W20260807 11:33:22.961515 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.008s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:33:23 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:33:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:33:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:33:23.286900 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38339
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:33:23.667182 10252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:23.667477 10252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:23.667615 10252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:23.675258 10252 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 11:33:23.675366 10252 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 11:33:23.675437 10252 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 11:33:23.675539 10252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:33:23.675629 10252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:23.675738 10252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:23.675799 10252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:33:23.675897 10252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:33:23.686208 10252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:23.687631 10252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:23.689050 10252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:23.701692 10257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:23.701603 10258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:23.701999 10252 server_base.cc:1034] running on GCE node
W20260807 11:33:23.701731 10260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:23.703210 10252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:23.704655 10252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:23.705909 10252 hybrid_clock.cc:648] HybridClock initialized: now 1786102403705856 us; error 35 us; skew 500 ppm
I20260807 11:33:23.706255 10252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:23 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102403, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:23.709635 10252 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:33:23.711130 10252 webserver.cc:460] Webserver started at http://127.24.103.126:39071/ using document root <none> and password file <none>
I20260807 11:33:23.711812 10252 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:23.711900 10252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:23.712198 10252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:33:23.715216 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "29a995ac34994ee08a6e36281f2aa3a4"
format_stamp: "Formatted at 2026-08-07 11:33:23 on dist-test-slave-wl2h"
server_key: "467439eb0947f2c02311d3c16ab8673c"
server_key_iv: "3df0753c2ced1a57e9518b477aab710b"
server_key_version: "encryptionkey@0"
I20260807 11:33:23.715972 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "29a995ac34994ee08a6e36281f2aa3a4"
format_stamp: "Formatted at 2026-08-07 11:33:23 on dist-test-slave-wl2h"
server_key: "467439eb0947f2c02311d3c16ab8673c"
server_key_iv: "3df0753c2ced1a57e9518b477aab710b"
server_key_version: "encryptionkey@0"
I20260807 11:33:23.721570 10252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:33:23.725328 10268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:23.726835 10252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:33:23.726989 10252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "29a995ac34994ee08a6e36281f2aa3a4"
format_stamp: "Formatted at 2026-08-07 11:33:23 on dist-test-slave-wl2h"
server_key: "467439eb0947f2c02311d3c16ab8673c"
server_key_iv: "3df0753c2ced1a57e9518b477aab710b"
server_key_version: "encryptionkey@0"
I20260807 11:33:23.727295 10252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:23.769349 10252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:23.774019 10252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:23.774272 10252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:23.794847 10252 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38339
I20260807 11:33:23.794867 10325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38339 every 8 connection(s)
I20260807 11:33:23.796298 10252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:33:23.800422 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 10252
I20260807 11:33:23.800563 10326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:23.800719 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:33:23.801435 24989 external_mini_cluster.cc:1428] Setting key 6c5e13c1236dd8ea093bf9eb40924d16
I20260807 11:33:23.816776 10326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Bootstrap starting.
Aug 07 11:33:23 dist-test-slave-wl2h krb5kdc[10239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:33:23.820303 10326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:23.821924 10326 log.cc:824] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:23.825793 10326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: No bootstrap required, opened a new log
I20260807 11:33:23.830287 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:23.803821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.24.103.126:38339)
0807 11:33:23.804761 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:23.804778 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:23.804834 (+ 56us) server_negotiation.cc:407] Connection header received
0807 11:33:23.808068 (+ 3234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:23.808102 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:23.808715 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:23.809235 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:23.810840 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:23.813165 (+ 2325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:23.814712 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:23.815895 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:23.816085 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:23.820747 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:23.820760 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:23.820789 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:23.820860 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:23.825315 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:23.826206 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:23.826216 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:23.826230 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:23.826459 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:23.826877 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:23.826884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:23.826889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:23.827648 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:23.827960 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:23.828379 (+ 419us) server_negotiation.cc:299] Negotiation successful
0807 11:33:23.828861 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":408,"threads_started":1}
I20260807 11:33:23.836159 10326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:33:23.836529 10326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:23.836601 10326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a995ac34994ee08a6e36281f2aa3a4, State: Initialized, Role: FOLLOWER
I20260807 11:33:23.837080 10326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:33:23.837278 10326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:33:23.837375 10326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:33:23.837499 10326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:23.839763 10326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:33:23.840209 10326 leader_election.cc:302] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29a995ac34994ee08a6e36281f2aa3a4; no voters:
I20260807 11:33:23.840772 10326 leader_election.cc:288] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:33:23.840932 10331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:33:23.841702 10331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 1 LEADER]: Becoming Leader. State: Replica: 29a995ac34994ee08a6e36281f2aa3a4, State: Running, Role: LEADER
I20260807 11:33:23.842324 10331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:33:23.843092 10326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:33:23.845465 10333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29a995ac34994ee08a6e36281f2aa3a4. Latest consensus state: current_term: 1 leader_uuid: "29a995ac34994ee08a6e36281f2aa3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } } }
I20260807 11:33:23.845912 10333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: This master's current role is: LEADER
I20260807 11:33:23.845471 10332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29a995ac34994ee08a6e36281f2aa3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } } }
I20260807 11:33:23.846666 10332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: This master's current role is: LEADER
I20260807 11:33:23.847728 10337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:33:23.853528 10337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:33:23.862640 10337 catalog_manager.cc:1347] Generated new cluster ID: a0d465aa9eaa4cac946a486e92385a2c
I20260807 11:33:23.862777 10337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:33:23.874603 10337 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:33:23.876366 10337 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:33:23.890156 10337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Generated new TSK 0
I20260807 11:33:23.891388 10337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:33:23.976164 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38339
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:33:24.328069 10355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:24.328449 10355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:24.328632 10355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:24.337749 10355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:24.337949 10355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:24.338178 10355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:33:24.350353 10355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:24.351918 10355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:24.353459 10355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:24.368311 10365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:24.368391 10363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:24.368652 10362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:24.369491 10355 server_base.cc:1034] running on GCE node
I20260807 11:33:24.370147 10355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:24.371265 10355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:24.372578 10355 hybrid_clock.cc:648] HybridClock initialized: now 1786102404372490 us; error 41 us; skew 500 ppm
I20260807 11:33:24.372974 10355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:24 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102404, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:24.376912 10355 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:33:24.378598 10355 webserver.cc:460] Webserver started at http://127.24.103.65:43597/ using document root <none> and password file <none>
I20260807 11:33:24.379448 10355 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:24.379619 10355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:24.379966 10355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:33:24.383345 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "e3e3da6e73154e4888a9a710f4c58f61"
format_stamp: "Formatted at 2026-08-07 11:33:24 on dist-test-slave-wl2h"
server_key: "328e255a285a9a94c796c45c91d606cc"
server_key_iv: "e0a1c0e9cd28e8dd08691bda087c210b"
server_key_version: "encryptionkey@0"
I20260807 11:33:24.384253 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "e3e3da6e73154e4888a9a710f4c58f61"
format_stamp: "Formatted at 2026-08-07 11:33:24 on dist-test-slave-wl2h"
server_key: "328e255a285a9a94c796c45c91d606cc"
server_key_iv: "e0a1c0e9cd28e8dd08691bda087c210b"
server_key_version: "encryptionkey@0"
I20260807 11:33:24.391165 10355 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 11:33:24.395637 10373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:24.397151 10355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260807 11:33:24.397356 10355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "e3e3da6e73154e4888a9a710f4c58f61"
format_stamp: "Formatted at 2026-08-07 11:33:24 on dist-test-slave-wl2h"
server_key: "328e255a285a9a94c796c45c91d606cc"
server_key_iv: "e0a1c0e9cd28e8dd08691bda087c210b"
server_key_version: "encryptionkey@0"
I20260807 11:33:24.397575 10355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:24.419412 10355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:24.424204 10355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:24.424563 10355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:24.425632 10355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:24.427182 10355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:24.427297 10355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:24.427462 10355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:24.427538 10355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:24.463289 10355 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:42241
I20260807 11:33:24.463341 10498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:42241 every 8 connection(s)
I20260807 11:33:24.464615 10355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:33:24.470989 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 10355
I20260807 11:33:24.471271 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:33:24.472199 24989 external_mini_cluster.cc:1428] Setting key 18a40f700270b0beedbcee76bbfc2ce6
Aug 07 11:33:24 dist-test-slave-wl2h krb5kdc[10239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102404, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:33:24.488044 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:24.467331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:42889 (local address 127.24.103.126:38339)
0807 11:33:24.467857 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:24.467861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:24.468676 (+ 815us) server_negotiation.cc:407] Connection header received
0807 11:33:24.470614 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:24.470618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:24.470707 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:24.470807 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:24.473020 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:24.473801 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:24.476064 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:24.476539 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:24.476657 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:24.481306 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:24.481312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:24.481316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:24.481355 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:24.484354 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:24.485267 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:24.485272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:24.485274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:24.485412 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:24.485932 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:24.485937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:24.485939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:24.486245 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:24.486442 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:24.487542 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0807 11:33:24.487744 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":279,"threads_started":1}
I20260807 11:33:24.489786 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:24.467738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:42889)
0807 11:33:24.468505 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:24.468554 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:24.470286 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:24.471145 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:24.471158 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:24.471961 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:33:24.472806 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:24.472832 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:24.473980 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:24.473985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:24.475877 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:24.475895 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:24.476809 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:24.476819 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:24.477169 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:24.477977 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:33:24.478028 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:33:24.481011 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:33:24.484559 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:24.484572 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:24.484594 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:24.485104 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:24.485553 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:24.485559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:24.485563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:24.485785 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:24.486399 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:33:24.486408 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:33:24.486974 (+ 566us) client_negotiation.cc:769] Sending connection context
0807 11:33:24.487397 (+ 423us) client_negotiation.cc:240] Negotiation successful
0807 11:33:24.488098 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":343,"threads_started":1}
I20260807 11:33:24.491647 10499 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38339
I20260807 11:33:24.491988 10499 heartbeater.cc:464] Registering TS with master...
I20260807 11:33:24.492830 10499 heartbeater.cc:511] Master 127.24.103.126:38339 requested a full tablet report, sending...
I20260807 11:33:24.495131 10286 ts_manager.cc:194] Registered new tserver with Master: e3e3da6e73154e4888a9a710f4c58f61 (127.24.103.65:42241)
I20260807 11:33:24.496791 10286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:42889
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:33:24.543946 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38339
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:33:24.896679 10507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:24.896977 10507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:24.897137 10507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:24.904681 10507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:24.904800 10507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:24.905009 10507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:33:24.915376 10507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:24.916760 10507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:24.918149 10507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:24.932387 10513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:24.932431 10515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:24.932370 10512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:24.933142 10507 server_base.cc:1034] running on GCE node
I20260807 11:33:24.933881 10507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:24.934875 10507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:24.936090 10507 hybrid_clock.cc:648] HybridClock initialized: now 1786102404936045 us; error 51 us; skew 500 ppm
I20260807 11:33:24.936396 10507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:24 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102404, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:24.939767 10507 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:33:24.941411 10507 webserver.cc:460] Webserver started at http://127.24.103.66:39641/ using document root <none> and password file <none>
I20260807 11:33:24.942129 10507 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:24.942288 10507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:24.942680 10507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:33:24.946025 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "72adee99138446e0b05e056656e36270"
format_stamp: "Formatted at 2026-08-07 11:33:24 on dist-test-slave-wl2h"
server_key: "ae50515a9b92654c6e2f6535a4402cb5"
server_key_iv: "19b5aac1d14793fe212f29657a6d5d80"
server_key_version: "encryptionkey@0"
I20260807 11:33:24.946762 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "72adee99138446e0b05e056656e36270"
format_stamp: "Formatted at 2026-08-07 11:33:24 on dist-test-slave-wl2h"
server_key: "ae50515a9b92654c6e2f6535a4402cb5"
server_key_iv: "19b5aac1d14793fe212f29657a6d5d80"
server_key_version: "encryptionkey@0"
I20260807 11:33:24.952265 10507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 11:33:24.955968 10523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:24.957420 10507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:33:24.957612 10507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "72adee99138446e0b05e056656e36270"
format_stamp: "Formatted at 2026-08-07 11:33:24 on dist-test-slave-wl2h"
server_key: "ae50515a9b92654c6e2f6535a4402cb5"
server_key_iv: "19b5aac1d14793fe212f29657a6d5d80"
server_key_version: "encryptionkey@0"
I20260807 11:33:24.957870 10507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:24.974421 10507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:24.979216 10507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:24.979506 10507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:24.980760 10507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:24.982221 10507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:24.982300 10507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:24.982388 10507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:24.982445 10507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:25.021219 10507 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:34113
I20260807 11:33:25.021351 10646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:34113 every 8 connection(s)
I20260807 11:33:25.022711 10507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:33:25.027524 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 10507
I20260807 11:33:25.027736 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:33:25.028228 24989 external_mini_cluster.cc:1428] Setting key 847a7b70b1b84f6644054f1f8e6a069f
Aug 07 11:33:25 dist-test-slave-wl2h krb5kdc[10239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102404, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:33:25.045009 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:25.025654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:52391 (local address 127.24.103.126:38339)
0807 11:33:25.026209 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:25.026215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:25.027287 (+ 1072us) server_negotiation.cc:407] Connection header received
0807 11:33:25.029472 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:25.029476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:25.029550 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:25.029671 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:25.031612 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.032562 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.034263 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.034728 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.034873 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:25.038601 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:25.038606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:25.038610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:25.038646 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:25.041480 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:25.042300 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:25.042304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:25.042306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:25.042438 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:25.042924 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:25.042931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:25.042934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:25.043367 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:25.043555 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:25.044330 (+ 775us) server_negotiation.cc:299] Negotiation successful
0807 11:33:25.044575 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":272,"threads_started":1}
I20260807 11:33:25.046077 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:25.026224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:52391)
0807 11:33:25.027115 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:25.027167 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:25.029161 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:25.029974 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:25.029987 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:25.030672 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:33:25.031402 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:25.031423 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.032747 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.032751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:25.034081 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:25.034093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.034991 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.034999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:25.035269 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:25.035995 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:33:25.036018 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:33:25.038353 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:33:25.041638 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:25.041648 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:25.041665 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:25.042115 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:25.042564 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:25.042570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:25.042574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:25.042791 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:25.043524 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:33:25.043533 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:33:25.043885 (+ 352us) client_negotiation.cc:769] Sending connection context
0807 11:33:25.044209 (+ 324us) client_negotiation.cc:240] Negotiation successful
0807 11:33:25.044820 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":325,"threads_started":1}
I20260807 11:33:25.047744 10647 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38339
I20260807 11:33:25.048149 10647 heartbeater.cc:464] Registering TS with master...
I20260807 11:33:25.048909 10647 heartbeater.cc:511] Master 127.24.103.126:38339 requested a full tablet report, sending...
I20260807 11:33:25.050559 10286 ts_manager.cc:194] Registered new tserver with Master: 72adee99138446e0b05e056656e36270 (127.24.103.66:34113)
I20260807 11:33:25.051914 10286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:52391
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:33:25.098754 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38339
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:33:25.440013 10653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:25.440291 10653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:25.440390 10653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:25.448344 10653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:25.448508 10653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:25.448783 10653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:33:25.462534 10653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:25.464010 10653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:25.465351 10653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:25.479084 10658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:25.479149 10661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:25.479184 10659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:25.479986 10653 server_base.cc:1034] running on GCE node
I20260807 11:33:25.480614 10653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:25.481694 10653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:25.482928 10653 hybrid_clock.cc:648] HybridClock initialized: now 1786102405482895 us; error 57 us; skew 500 ppm
I20260807 11:33:25.483296 10653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:25 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102405, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:25.486555 10653 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:33:25.488153 10653 webserver.cc:460] Webserver started at http://127.24.103.67:35107/ using document root <none> and password file <none>
I20260807 11:33:25.488844 10653 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:25.488992 10653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:25.489329 10653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:33:25.492375 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "3ebf7c502b114b1785348f2ae12917f1"
format_stamp: "Formatted at 2026-08-07 11:33:25 on dist-test-slave-wl2h"
server_key: "96df2dd6e8eb857b6475c3311d42fb2c"
server_key_iv: "3983ce24c4a48aa97e7a1038683c3055"
server_key_version: "encryptionkey@0"
I20260807 11:33:25.493135 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "3ebf7c502b114b1785348f2ae12917f1"
format_stamp: "Formatted at 2026-08-07 11:33:25 on dist-test-slave-wl2h"
server_key: "96df2dd6e8eb857b6475c3311d42fb2c"
server_key_iv: "3983ce24c4a48aa97e7a1038683c3055"
server_key_version: "encryptionkey@0"
I20260807 11:33:25.498284 10653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 11:33:25.500111 10499 heartbeater.cc:503] Master 127.24.103.126:38339 was elected leader, sending a full tablet report...
I20260807 11:33:25.502144 10669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:25.503566 10653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:33:25.503741 10653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "3ebf7c502b114b1785348f2ae12917f1"
format_stamp: "Formatted at 2026-08-07 11:33:25 on dist-test-slave-wl2h"
server_key: "96df2dd6e8eb857b6475c3311d42fb2c"
server_key_iv: "3983ce24c4a48aa97e7a1038683c3055"
server_key_version: "encryptionkey@0"
I20260807 11:33:25.503942 10653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:25.539381 10653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:25.544129 10653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:25.544432 10653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:25.545486 10653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:25.546883 10653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:25.546957 10653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:25.547125 10653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:25.547178 10653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:25.584062 10653 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:39911
I20260807 11:33:25.584162 10792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:39911 every 8 connection(s)
I20260807 11:33:25.585424 10653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:33:25.589895 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 10653
I20260807 11:33:25.590104 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:33:25.590757 24989 external_mini_cluster.cc:1428] Setting key bcf507fcc2c1af514e5fe91b3768d106
Aug 07 11:33:25 dist-test-slave-wl2h krb5kdc[10239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102405, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:33:25.607606 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:25.588113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:40655 (local address 127.24.103.126:38339)
0807 11:33:25.588606 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:25.588610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:25.589568 (+ 958us) server_negotiation.cc:407] Connection header received
0807 11:33:25.592263 (+ 2695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:25.592269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:25.592368 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:25.592466 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:25.594333 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.595332 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.596842 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.597297 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.597402 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:25.600943 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:25.600948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:25.600952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:25.600988 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:25.604010 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:25.604903 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:25.604907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:25.604910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:25.605068 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:25.605627 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:25.605631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:25.605633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:25.606049 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:25.606220 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:25.607085 (+ 865us) server_negotiation.cc:299] Negotiation successful
0807 11:33:25.607300 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":238,"threads_started":1}
I20260807 11:33:25.608675 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:25.588645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:40655)
0807 11:33:25.589346 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:25.589395 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:25.591864 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:25.592764 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:25.592778 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:25.593440 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:33:25.594125 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:25.594153 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.595489 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.595493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:25.596705 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:25.596717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.597494 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.597499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:25.597690 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:25.598367 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:33:25.598392 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:33:25.600736 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:33:25.604151 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:25.604158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:25.604170 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:25.604744 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:25.605194 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:25.605198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:25.605200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:25.605486 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:25.606183 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:33:25.606189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:33:25.606643 (+ 454us) client_negotiation.cc:769] Sending connection context
0807 11:33:25.606939 (+ 296us) client_negotiation.cc:240] Negotiation successful
0807 11:33:25.607435 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":270,"threads_started":1}
I20260807 11:33:25.610247 10793 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38339
I20260807 11:33:25.610582 10793 heartbeater.cc:464] Registering TS with master...
I20260807 11:33:25.611384 10793 heartbeater.cc:511] Master 127.24.103.126:38339 requested a full tablet report, sending...
I20260807 11:33:25.612967 10286 ts_manager.cc:194] Registered new tserver with Master: 3ebf7c502b114b1785348f2ae12917f1 (127.24.103.67:39911)
I20260807 11:33:25.613772 10286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:40655
I20260807 11:33:25.618975 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:33:25.637250 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:25.623728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.24.103.126:38339)
0807 11:33:25.623951 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:25.623956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:25.624459 (+ 503us) server_negotiation.cc:407] Connection header received
0807 11:33:25.624614 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:25.624618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:25.624692 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:25.624786 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:25.626233 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.627104 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.628559 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.629036 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.629138 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:25.631807 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:25.631812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:25.631815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:25.631871 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:25.634625 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:25.635425 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:25.635429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:25.635431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:25.635552 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:25.635898 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:25.635901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:25.635903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:25.636282 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:25.636457 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:25.636788 (+ 331us) server_negotiation.cc:299] Negotiation successful
0807 11:33:25.636973 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 11:33:25.641546 10286 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:33:25.648881 10286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:33:25.675979 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:25.661986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34113 (local address 127.0.0.1:57880)
0807 11:33:25.663302 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:25.663365 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:25.663586 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:25.665042 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:25.665055 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:25.665111 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:25.665619 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:25.665635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.668410 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.668420 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:25.671789 (+ 3369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:25.671814 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.674564 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.674571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:25.674798 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:25.674870 (+ 72us) client_negotiation.cc:769] Sending connection context
0807 11:33:25.675000 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 11:33:25.675369 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":1048,"threads_started":1}
I20260807 11:33:25.677922 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:25.662201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.24.103.66:34113)
0807 11:33:25.664221 (+ 2020us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:25.664234 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:25.664275 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:33:25.664406 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:25.664419 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:25.664745 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:25.665845 (+ 1100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:25.666104 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.668177 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.672127 (+ 3950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.674278 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.676727 (+ 2449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:25.677066 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:25.677165 (+ 99us) server_negotiation.cc:299] Negotiation successful
0807 11:33:25.677441 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1844,"thread_start_us":1724,"threads_started":1}
I20260807 11:33:25.678457 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:25.662183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:39911 (local address 127.0.0.1:39970)
0807 11:33:25.664217 (+ 2034us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:25.664236 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:25.664348 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:25.666159 (+ 1811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:25.666166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:25.666202 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:25.666768 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:25.666783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.669572 (+ 2789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.669580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:25.674255 (+ 4675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:25.674278 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.677289 (+ 3011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.677299 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:25.677571 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:25.677637 (+ 66us) client_negotiation.cc:769] Sending connection context
0807 11:33:25.677752 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 11:33:25.677955 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1909,"thread_start_us":1754,"threads_started":1}
I20260807 11:33:25.679159 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:25.662823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.24.103.67:39911)
0807 11:33:25.665138 (+ 2315us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:25.665148 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:25.665186 (+ 38us) server_negotiation.cc:407] Connection header received
0807 11:33:25.665300 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:25.665311 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:25.665579 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:25.665817 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:25.667121 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.669325 (+ 2204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.674531 (+ 5206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.676607 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.676790 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:25.677099 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:25.678404 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0807 11:33:25.678673 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2139,"thread_start_us":1963,"threads_started":1}
I20260807 11:33:25.683738 10725 tablet_service.cc:1467] Processing CreateTablet for tablet e03e8db4ab9e47c88210e4eb4c7877f6 (DEFAULT_TABLE table=test-table [id=a6e8063f3a8b4abd9ebe58c5d6e7f118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:25.683738 10573 tablet_service.cc:1467] Processing CreateTablet for tablet e03e8db4ab9e47c88210e4eb4c7877f6 (DEFAULT_TABLE table=test-table [id=a6e8063f3a8b4abd9ebe58c5d6e7f118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:25.685355 10725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03e8db4ab9e47c88210e4eb4c7877f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:25.685355 10573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03e8db4ab9e47c88210e4eb4c7877f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:25.687098 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:25.664794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42241 (local address 127.0.0.1:47132)
0807 11:33:25.676314 (+ 11520us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:25.676335 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:25.676475 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:25.681029 (+ 4554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:25.681034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:25.681064 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:25.681256 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:25.681265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.683073 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.683080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:25.684399 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:25.684412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.686160 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.686167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:25.686375 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:25.686418 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 11:33:25.686553 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 11:33:25.686720 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11418,"thread_start_us":14769,"threads_started":1}
I20260807 11:33:25.687173 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:25.665632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.24.103.65:42241)
0807 11:33:25.680308 (+ 14676us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:25.680318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:25.680350 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:33:25.680457 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:25.680466 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:25.680763 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:25.681411 (+ 648us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:25.681694 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.682874 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.684660 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:25.685924 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:25.686096 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:25.686348 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:25.686610 (+ 262us) server_negotiation.cc:299] Negotiation successful
0807 11:33:25.686780 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14463,"thread_start_us":15267,"threads_started":1}
I20260807 11:33:25.690649 10428 tablet_service.cc:1467] Processing CreateTablet for tablet e03e8db4ab9e47c88210e4eb4c7877f6 (DEFAULT_TABLE table=test-table [id=a6e8063f3a8b4abd9ebe58c5d6e7f118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:25.692032 10428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03e8db4ab9e47c88210e4eb4c7877f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:25.700651 10819 tablet_bootstrap.cc:492] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: Bootstrap starting.
I20260807 11:33:25.701601 10820 tablet_bootstrap.cc:492] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: Bootstrap starting.
I20260807 11:33:25.704931 10820 tablet_bootstrap.cc:654] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:25.705106 10819 tablet_bootstrap.cc:654] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:25.706210 10820 log.cc:824] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:25.706404 10821 tablet_bootstrap.cc:492] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: Bootstrap starting.
I20260807 11:33:25.706509 10819 log.cc:824] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:25.711215 10821 tablet_bootstrap.cc:654] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:25.712639 10821 log.cc:824] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:25.714213 10819 tablet_bootstrap.cc:492] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: No bootstrap required, opened a new log
I20260807 11:33:25.714255 10820 tablet_bootstrap.cc:492] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: No bootstrap required, opened a new log
I20260807 11:33:25.714563 10820 ts_tablet_manager.cc:1397] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260807 11:33:25.714586 10819 ts_tablet_manager.cc:1397] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260807 11:33:25.716581 10821 tablet_bootstrap.cc:492] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: No bootstrap required, opened a new log
I20260807 11:33:25.716893 10821 ts_tablet_manager.cc:1397] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260807 11:33:25.722930 10820 raft_consensus.cc:348] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } } peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } }
I20260807 11:33:25.723443 10820 raft_consensus.cc:374] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:25.723590 10820 raft_consensus.cc:729] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72adee99138446e0b05e056656e36270, State: Initialized, Role: FOLLOWER
I20260807 11:33:25.724260 10820 consensus_queue.cc:260] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } } peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } }
I20260807 11:33:25.725455 10647 heartbeater.cc:503] Master 127.24.103.126:38339 was elected leader, sending a full tablet report...
I20260807 11:33:25.725540 10819 raft_consensus.cc:348] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } } peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } }
I20260807 11:33:25.725999 10819 raft_consensus.cc:374] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:25.726091 10819 raft_consensus.cc:729] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ebf7c502b114b1785348f2ae12917f1, State: Initialized, Role: FOLLOWER
I20260807 11:33:25.726235 10820 ts_tablet_manager.cc:1428] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260807 11:33:25.726714 10819 consensus_queue.cc:260] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } } peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } }
I20260807 11:33:25.728169 10821 raft_consensus.cc:348] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } } peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } }
I20260807 11:33:25.728686 10821 raft_consensus.cc:374] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:25.728729 10793 heartbeater.cc:503] Master 127.24.103.126:38339 was elected leader, sending a full tablet report...
I20260807 11:33:25.728840 10821 raft_consensus.cc:729] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3e3da6e73154e4888a9a710f4c58f61, State: Initialized, Role: FOLLOWER
I20260807 11:33:25.729519 10819 ts_tablet_manager.cc:1428] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260807 11:33:25.729502 10821 consensus_queue.cc:260] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } } peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } }
I20260807 11:33:25.731372 10821 ts_tablet_manager.cc:1428] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260807 11:33:25.775236 10648 tablet.cc:2369] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:33:25.837150 10794 tablet.cc:2369] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:33:25.967372 10501 tablet.cc:2369] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:33:26.047359 10828 raft_consensus.cc:482] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:33:26.047731 10828 raft_consensus.cc:504] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } } peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } }
I20260807 11:33:26.049613 10828 leader_election.cc:288] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72adee99138446e0b05e056656e36270 (127.24.103.66:34113), 3ebf7c502b114b1785348f2ae12917f1 (127.24.103.67:39911)
I20260807 11:33:26.057112 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:26.050193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:46717 (local address 127.24.103.67:39911)
0807 11:33:26.050427 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:26.050437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:26.051311 (+ 874us) server_negotiation.cc:407] Connection header received
0807 11:33:26.051545 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:26.051557 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:26.051625 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:26.051729 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:26.052394 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.053178 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.055242 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.056018 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.056146 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:26.056199 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:26.056694 (+ 495us) server_negotiation.cc:299] Negotiation successful
0807 11:33:26.056832 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 11:33:26.057255 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:26.050178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:57605 (local address 127.24.103.66:34113)
0807 11:33:26.050442 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:26.050451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:26.051373 (+ 922us) server_negotiation.cc:407] Connection header received
0807 11:33:26.051554 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:26.051558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:26.051623 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:26.051734 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:26.052508 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.053349 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.055269 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.056210 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.056378 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:26.056459 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:26.056751 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 11:33:26.056943 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:33:26.057294 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:26.050304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34113 (local address 127.24.103.65:57605)
0807 11:33:26.051216 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:26.051231 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:26.051406 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:26.051751 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:26.051756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:26.051801 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:26.052298 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:26.052332 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.053495 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.053499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:26.055004 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:26.055075 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.056436 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.056441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:26.056555 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:26.056577 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 11:33:26.056694 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 11:33:26.056824 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":3,"spinlock_wait_cycles":33920,"thread_start_us":522,"threads_started":1}
I20260807 11:33:26.057400 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:26.050033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:39911 (local address 127.24.103.65:46717)
0807 11:33:26.051166 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:26.051196 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:26.051384 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:26.051753 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:26.051757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:26.051782 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:26.052233 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:26.052246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.053331 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.053336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:26.055077 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:26.055091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.056259 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.056269 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:26.056486 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:26.056533 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 11:33:26.056617 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 11:33:26.056826 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":34,"thread_start_us":626,"threads_started":1}
I20260807 11:33:26.058753 10746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03e8db4ab9e47c88210e4eb4c7877f6" candidate_uuid: "e3e3da6e73154e4888a9a710f4c58f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ebf7c502b114b1785348f2ae12917f1" is_pre_election: true
I20260807 11:33:26.058753 10596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03e8db4ab9e47c88210e4eb4c7877f6" candidate_uuid: "e3e3da6e73154e4888a9a710f4c58f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72adee99138446e0b05e056656e36270" is_pre_election: true
I20260807 11:33:26.059404 10746 raft_consensus.cc:2454] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3e3da6e73154e4888a9a710f4c58f61 in term 0.
I20260807 11:33:26.059415 10596 raft_consensus.cc:2454] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3e3da6e73154e4888a9a710f4c58f61 in term 0.
I20260807 11:33:26.060285 10374 leader_election.cc:302] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72adee99138446e0b05e056656e36270, e3e3da6e73154e4888a9a710f4c58f61; no voters:
I20260807 11:33:26.060797 10828 raft_consensus.cc:2781] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:33:26.061002 10828 raft_consensus.cc:482] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:33:26.061192 10828 raft_consensus.cc:3037] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:26.063757 10828 raft_consensus.cc:504] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } } peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } }
I20260807 11:33:26.064522 10828 leader_election.cc:288] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [CANDIDATE]: Term 1 election: Requested vote from peers 72adee99138446e0b05e056656e36270 (127.24.103.66:34113), 3ebf7c502b114b1785348f2ae12917f1 (127.24.103.67:39911)
I20260807 11:33:26.065048 10596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03e8db4ab9e47c88210e4eb4c7877f6" candidate_uuid: "e3e3da6e73154e4888a9a710f4c58f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72adee99138446e0b05e056656e36270"
I20260807 11:33:26.065078 10746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03e8db4ab9e47c88210e4eb4c7877f6" candidate_uuid: "e3e3da6e73154e4888a9a710f4c58f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ebf7c502b114b1785348f2ae12917f1"
I20260807 11:33:26.065263 10596 raft_consensus.cc:3037] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:26.065341 10746 raft_consensus.cc:3037] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:26.067683 10596 raft_consensus.cc:2454] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3e3da6e73154e4888a9a710f4c58f61 in term 1.
I20260807 11:33:26.067900 10746 raft_consensus.cc:2454] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3e3da6e73154e4888a9a710f4c58f61 in term 1.
I20260807 11:33:26.068234 10374 leader_election.cc:302] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72adee99138446e0b05e056656e36270, e3e3da6e73154e4888a9a710f4c58f61; no voters:
I20260807 11:33:26.068567 10828 raft_consensus.cc:2781] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:33:26.069027 10828 raft_consensus.cc:686] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 1 LEADER]: Becoming Leader. State: Replica: e3e3da6e73154e4888a9a710f4c58f61, State: Running, Role: LEADER
I20260807 11:33:26.069588 10828 consensus_queue.cc:237] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } } peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } }
I20260807 11:33:26.074337 10286 catalog_manager.cc:5495] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3e3da6e73154e4888a9a710f4c58f61 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "e3e3da6e73154e4888a9a710f4c58f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } health_report { overall_health: HEALTHY } } }
I20260807 11:33:26.133266 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:26.127510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.24.103.65:42241)
0807 11:33:26.127748 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:26.127752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:26.127771 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:33:26.127972 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:26.127977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:26.128124 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:26.128263 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:33:26.128633 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.129381 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.130835 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.131399 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.131544 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:26.132075 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:33:26.132603 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:33:26.132776 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:26.132890 (+ 114us) server_negotiation.cc:299] Negotiation successful
0807 11:33:26.132975 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 11:33:26.141702 10596 raft_consensus.cc:1261] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270 [term 1 FOLLOWER]: Refusing update from remote peer e3e3da6e73154e4888a9a710f4c58f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:33:26.141698 10746 raft_consensus.cc:1261] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1 [term 1 FOLLOWER]: Refusing update from remote peer e3e3da6e73154e4888a9a710f4c58f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:33:26.142817 10831 consensus_queue.cc:1035] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:33:26.143350 10828 consensus_queue.cc:1035] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:33:26.154917 10834 mvcc.cc:205] Tried to move back new op lower bound from 7315875455545733120 to 7315875455264575488. Current Snapshot: MvccSnapshot[applied={T|T < 7315875455545733120}]
I20260807 11:33:26.159129 10837 mvcc.cc:205] Tried to move back new op lower bound from 7315875455545733120 to 7315875455264575488. Current Snapshot: MvccSnapshot[applied={T|T < 7315875455545733120}]
I20260807 11:33:26.169000 10285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:33:26.169307 10285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:33:26.173543 10285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Sending DeleteTablet for 3 replicas of tablet e03e8db4ab9e47c88210e4eb4c7877f6
I20260807 11:33:26.175066 10725 tablet_service.cc:1514] Processing DeleteTablet for tablet e03e8db4ab9e47c88210e4eb4c7877f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:26 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:39970
I20260807 11:33:26.175087 10573 tablet_service.cc:1514] Processing DeleteTablet for tablet e03e8db4ab9e47c88210e4eb4c7877f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:26 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:57880
I20260807 11:33:26.175382 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 10355
I20260807 11:33:26.176539 10848 tablet_replica.cc:330] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: stopping tablet replica
I20260807 11:33:26.176908 10428 tablet_service.cc:1514] Processing DeleteTablet for tablet e03e8db4ab9e47c88210e4eb4c7877f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:26 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:47132
I20260807 11:33:26.177037 10848 raft_consensus.cc:2229] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:33:26.177887 10848 raft_consensus.cc:2258] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:26.178153 10850 tablet_replica.cc:330] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: stopping tablet replica
I20260807 11:33:26.178831 10847 tablet_replica.cc:330] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: stopping tablet replica
I20260807 11:33:26.179069 10850 raft_consensus.cc:2229] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:33:26.179335 10847 raft_consensus.cc:2229] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:33:26.180078 10850 raft_consensus.cc:2258] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:26.180135 10847 raft_consensus.cc:2258] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:26.182461 10848 ts_tablet_manager.cc:1905] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:26.182705 10850 ts_tablet_manager.cc:1905] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:26.185498 10847 ts_tablet_manager.cc:1905] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:26.190485 10848 ts_tablet_manager.cc:1918] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:26.190752 10848 log.cc:1196] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/e03e8db4ab9e47c88210e4eb4c7877f6
I20260807 11:33:26.191475 10848 ts_tablet_manager.cc:1939] T e03e8db4ab9e47c88210e4eb4c7877f6 P 3ebf7c502b114b1785348f2ae12917f1: Deleting consensus metadata
I20260807 11:33:26.191480 10850 ts_tablet_manager.cc:1918] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:26.191722 10850 log.cc:1196] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/e03e8db4ab9e47c88210e4eb4c7877f6
I20260807 11:33:26.192359 10850 ts_tablet_manager.cc:1939] T e03e8db4ab9e47c88210e4eb4c7877f6 P e3e3da6e73154e4888a9a710f4c58f61: Deleting consensus metadata
I20260807 11:33:26.193814 10271 catalog_manager.cc:4841] TS 3ebf7c502b114b1785348f2ae12917f1 (127.24.103.67:39911): tablet e03e8db4ab9e47c88210e4eb4c7877f6 (table test-table [id=a6e8063f3a8b4abd9ebe58c5d6e7f118]) successfully deleted
I20260807 11:33:26.194146 10269 catalog_manager.cc:4841] TS e3e3da6e73154e4888a9a710f4c58f61 (127.24.103.65:42241): tablet e03e8db4ab9e47c88210e4eb4c7877f6 (table test-table [id=a6e8063f3a8b4abd9ebe58c5d6e7f118]) successfully deleted
I20260807 11:33:26.194906 10847 ts_tablet_manager.cc:1918] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:26.195122 10847 log.cc:1196] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/e03e8db4ab9e47c88210e4eb4c7877f6
I20260807 11:33:26.195591 10847 ts_tablet_manager.cc:1939] T e03e8db4ab9e47c88210e4eb4c7877f6 P 72adee99138446e0b05e056656e36270: Deleting consensus metadata
I20260807 11:33:26.197139 10269 catalog_manager.cc:4841] TS 72adee99138446e0b05e056656e36270 (127.24.103.66:34113): tablet e03e8db4ab9e47c88210e4eb4c7877f6 (table test-table [id=a6e8063f3a8b4abd9ebe58c5d6e7f118]) successfully deleted
Aug 07 11:33:26 dist-test-slave-wl2h krb5kdc[10239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:33:26.214355 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:26.176447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.24.103.65:42241)
0807 11:33:26.176672 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:26.176682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:26.196362 (+ 19680us) server_negotiation.cc:407] Connection header received
0807 11:33:26.196577 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:26.196583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:26.196704 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:26.196825 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:26.198892 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.199854 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.201840 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.202418 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.202519 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:26.207999 (+ 5480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:26.208019 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:26.208048 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:26.208090 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:26.211363 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:26.212122 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:26.212129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:26.212132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:26.212293 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:26.212700 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:26.212705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:26.212708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:26.213086 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:26.213408 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:26.213740 (+ 332us) server_negotiation.cc:299] Negotiation successful
0807 11:33:26.213986 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:33:26.215186 10494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:26.315683 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 10355
I20260807 11:33:26.339259 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 10507
Aug 07 11:33:26 dist-test-slave-wl2h krb5kdc[10239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:33:26.355320 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:26.340186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.24.103.66:34113)
0807 11:33:26.340406 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:26.340411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:26.340432 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:33:26.340491 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:26.340495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:26.340558 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:26.340649 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:26.342029 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.342749 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.344338 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.344740 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.344844 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:26.349551 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:26.349565 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:26.349581 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:26.349620 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:26.352589 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:26.353274 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:26.353280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:26.353283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:26.353407 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:26.353787 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:26.353791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:26.353793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:26.354217 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:26.354448 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:26.354716 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 11:33:26.354885 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 11:33:26.356004 10642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:26.448016 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 10507
I20260807 11:33:26.468974 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 10653
Aug 07 11:33:26 dist-test-slave-wl2h krb5kdc[10239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:33:26.492321 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:26.469855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.24.103.67:39911)
0807 11:33:26.470062 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:26.470066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:26.470111 (+ 45us) server_negotiation.cc:407] Connection header received
0807 11:33:26.470310 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:26.470315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:26.470387 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:26.470494 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:26.472206 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.472956 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.474524 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:26.474971 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:26.475313 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:26.485932 (+ 10619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:26.485949 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:26.485966 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:26.486058 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:26.489144 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:26.489998 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:26.490006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:26.490009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:26.490221 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:26.490630 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:26.490634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:26.490636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:26.491229 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:26.491482 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:26.491813 (+ 331us) server_negotiation.cc:299] Negotiation successful
0807 11:33:26.492001 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 11:33:26.493032 10788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:26.586638 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 10653
I20260807 11:33:26.607156 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 10252
I20260807 11:33:26.608410 10320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:26.725937 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 10252
I20260807 11:33:46.752928 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38339
--webserver_interface=127.24.103.126
--webserver_port=39071
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:33:47.115864 10855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:47.116250 10855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:47.116434 10855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:47.124235 10855 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 11:33:47.124382 10855 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 11:33:47.124476 10855 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 11:33:47.124586 10855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:33:47.124696 10855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:47.124800 10855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:47.124890 10855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:33:47.124970 10855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:33:47.136857 10855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=39071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:47.138423 10855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:47.139995 10855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:47.168300 10866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:47.168411 10863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:47.168407 10864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:47.169637 10855 server_base.cc:1034] running on GCE node
I20260807 11:33:47.170621 10855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:47.172489 10855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:47.173750 10855 hybrid_clock.cc:648] HybridClock initialized: now 1786102427173656 us; error 60 us; skew 500 ppm
I20260807 11:33:47.174212 10855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:47 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102427, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:47.178108 10855 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:33:47.180128 10855 webserver.cc:460] Webserver started at http://127.24.103.126:39071/ using document root <none> and password file <none>
I20260807 11:33:47.180961 10855 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:47.181078 10855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:47.187721 10855 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:33:47.190791 10874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:47.192495 10855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:33:47.192737 10855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "29a995ac34994ee08a6e36281f2aa3a4"
format_stamp: "Formatted at 2026-08-07 11:33:23 on dist-test-slave-wl2h"
server_key: "467439eb0947f2c02311d3c16ab8673c"
server_key_iv: "3df0753c2ced1a57e9518b477aab710b"
server_key_version: "encryptionkey@0"
I20260807 11:33:47.193845 10855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:47.209923 10855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:47.214802 10855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:47.215227 10855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:47.236850 10855 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38339
I20260807 11:33:47.236892 10931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38339 every 8 connection(s)
I20260807 11:33:47.238518 10855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:33:47.246366 10932 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:33:47.247098 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 10855
I20260807 11:33:47.247957 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:42241
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:38339
--webserver_port=43597
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:33:47.250047 10932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Bootstrap starting.
I20260807 11:33:47.270646 10932 log.cc:824] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:47.283469 10932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:33:47.284060 10932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Bootstrap complete.
I20260807 11:33:47.293723 10932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:33:47.294647 10932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a995ac34994ee08a6e36281f2aa3a4, State: Initialized, Role: FOLLOWER
I20260807 11:33:47.295354 10932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:33:47.295617 10932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:33:47.295769 10932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:33:47.295946 10932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 1 FOLLOWER]: Advancing to term 2
I20260807 11:33:47.300190 10932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:33:47.300746 10932 leader_election.cc:302] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29a995ac34994ee08a6e36281f2aa3a4; no voters:
I20260807 11:33:47.301497 10932 leader_election.cc:288] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 11:33:47.301644 10937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 11:33:47.304222 10932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:33:47.308881 10937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 2 LEADER]: Becoming Leader. State: Replica: 29a995ac34994ee08a6e36281f2aa3a4, State: Running, Role: LEADER
I20260807 11:33:47.309880 10937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:33:47.316685 10938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "29a995ac34994ee08a6e36281f2aa3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } } }
I20260807 11:33:47.317237 10938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: This master's current role is: LEADER
I20260807 11:33:47.317113 10939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29a995ac34994ee08a6e36281f2aa3a4. Latest consensus state: current_term: 2 leader_uuid: "29a995ac34994ee08a6e36281f2aa3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } } }
I20260807 11:33:47.317468 10939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: This master's current role is: LEADER
I20260807 11:33:47.327258 10953 catalog_manager.cc:1259] Loaded cluster ID: a0d465aa9eaa4cac946a486e92385a2c
I20260807 11:33:47.327373 10953 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: loading cluster ID for follower catalog manager: success
I20260807 11:33:47.331596 10953 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: acquiring CA information for follower catalog manager: success
I20260807 11:33:47.334408 10953 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:33:47.334612 10954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:33:47.339382 10954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:33:47.340634 10954 catalog_manager.cc:1259] Loaded cluster ID: a0d465aa9eaa4cac946a486e92385a2c
I20260807 11:33:47.340739 10954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:33:47.342480 10954 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:33:47.343750 10954 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Loaded TSK: 0
I20260807 11:33:47.372143 10954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Generated new TSK 1
I20260807 11:33:47.372550 10954 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 11:33:47.617595 10934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:47.617946 10934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:47.618161 10934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:47.626009 10934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:47.626183 10934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:47.626362 10934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:33:47.636790 10934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:42241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=43597
--tserver_master_addrs=127.24.103.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:47.638221 10934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:47.639853 10934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:47.655040 10963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:47.655694 10967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:47.655407 10964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:47.656575 10934 server_base.cc:1034] running on GCE node
I20260807 11:33:47.657388 10934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:47.658660 10934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:47.659926 10934 hybrid_clock.cc:648] HybridClock initialized: now 1786102427659865 us; error 49 us; skew 500 ppm
I20260807 11:33:47.660306 10934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:47 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102427, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:47.664399 10934 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:33:47.666316 10934 webserver.cc:460] Webserver started at http://127.24.103.65:43597/ using document root <none> and password file <none>
I20260807 11:33:47.667276 10934 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:47.667446 10934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:47.675171 10934 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:33:47.678640 10974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:47.680420 10934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:33:47.680594 10934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "e3e3da6e73154e4888a9a710f4c58f61"
format_stamp: "Formatted at 2026-08-07 11:33:24 on dist-test-slave-wl2h"
server_key: "328e255a285a9a94c796c45c91d606cc"
server_key_iv: "e0a1c0e9cd28e8dd08691bda087c210b"
server_key_version: "encryptionkey@0"
I20260807 11:33:47.681951 10934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:47.707994 10934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:47.713020 10934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:47.713348 10934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:47.714483 10934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:47.716225 10934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:47.716368 10934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:47.716516 10934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:47.716622 10934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:47.753991 10934 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:42241
I20260807 11:33:47.754086 11101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:42241 every 8 connection(s)
I20260807 11:33:47.755538 10934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:33:47.761284 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 10934
I20260807 11:33:47.761966 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:34113
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:38339
--webserver_port=39641
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:33:47 dist-test-slave-wl2h krb5kdc[10239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102427, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:33:47.780246 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:47.758916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:51723)
0807 11:33:47.759805 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:47.759859 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:47.761665 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:47.763279 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:47.763291 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:47.763903 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:33:47.764410 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:47.764427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:47.766061 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:47.766072 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:47.767447 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:47.767460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:47.768481 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:47.768489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:47.768767 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:47.769477 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:33:47.769504 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:33:47.772051 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:33:47.776032 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:47.776039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:47.776049 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:47.776482 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:47.777025 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:47.777029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:47.777031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:47.777235 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:47.778235 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:33:47.778241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:33:47.778525 (+ 284us) client_negotiation.cc:769] Sending connection context
0807 11:33:47.778762 (+ 237us) client_negotiation.cc:240] Negotiation successful
0807 11:33:47.779244 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":351,"threads_started":1}
I20260807 11:33:47.780746 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:47.758677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:51723 (local address 127.24.103.126:38339)
0807 11:33:47.759389 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:47.759401 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:47.760012 (+ 611us) server_negotiation.cc:407] Connection header received
0807 11:33:47.762242 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:47.762258 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:47.762697 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:47.763135 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:47.764609 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:47.765868 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:47.767604 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:47.768224 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:47.768437 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:47.772346 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:47.772354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:47.772367 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:47.772408 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:47.775872 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:47.776640 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:47.776651 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:47.776661 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:47.776883 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:47.777376 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:47.777384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:47.777390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:47.778078 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:47.778282 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:47.779089 (+ 807us) server_negotiation.cc:299] Negotiation successful
0807 11:33:47.779538 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":304,"threads_started":1}
I20260807 11:33:47.783474 11102 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38339
I20260807 11:33:47.783798 11102 heartbeater.cc:464] Registering TS with master...
I20260807 11:33:47.784533 11102 heartbeater.cc:511] Master 127.24.103.126:38339 requested a full tablet report, sending...
I20260807 11:33:47.786909 10893 ts_manager.cc:194] Registered new tserver with Master: e3e3da6e73154e4888a9a710f4c58f61 (127.24.103.65:42241)
I20260807 11:33:47.788462 10893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:51723
W20260807 11:33:48.113253 11106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:48.113585 11106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:48.113760 11106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:48.121217 11106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:48.121368 11106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:48.121536 11106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:33:48.131320 11106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:34113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=39641
--tserver_master_addrs=127.24.103.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:48.132772 11106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:48.134096 11106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:48.148102 11113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:48.148307 11112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:48.148146 11117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:48.149717 11106 server_base.cc:1034] running on GCE node
I20260807 11:33:48.150393 11106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:48.151474 11106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:48.152784 11106 hybrid_clock.cc:648] HybridClock initialized: now 1786102428152720 us; error 68 us; skew 500 ppm
I20260807 11:33:48.153151 11106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:48 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102428, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:48.156685 11106 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:33:48.158277 11106 webserver.cc:460] Webserver started at http://127.24.103.66:39641/ using document root <none> and password file <none>
I20260807 11:33:48.159003 11106 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:48.159219 11106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:48.164834 11106 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:33:48.167693 11124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:48.169327 11106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:33:48.169533 11106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "72adee99138446e0b05e056656e36270"
format_stamp: "Formatted at 2026-08-07 11:33:24 on dist-test-slave-wl2h"
server_key: "ae50515a9b92654c6e2f6535a4402cb5"
server_key_iv: "19b5aac1d14793fe212f29657a6d5d80"
server_key_version: "encryptionkey@0"
I20260807 11:33:48.170610 11106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:48.193317 11106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:48.197855 11106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:48.198150 11106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:48.199357 11106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:48.200757 11106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:48.200881 11106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:48.201020 11106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:48.201116 11106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:48.235237 11106 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:34113
I20260807 11:33:48.235332 11250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:34113 every 8 connection(s)
I20260807 11:33:48.237076 11106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:33:48.247485 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 11106
I20260807 11:33:48.248266 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:39911
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:38339
--webserver_port=35107
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 11:33:48 dist-test-slave-wl2h krb5kdc[10239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102428, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:33:48.258462 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.240025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:34969 (local address 127.24.103.126:38339)
0807 11:33:48.240232 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:48.240237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:48.241527 (+ 1290us) server_negotiation.cc:407] Connection header received
0807 11:33:48.243485 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:48.243491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:48.243600 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:48.243716 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:48.245184 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.245895 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.247704 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.248169 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.248246 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.252152 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:48.252157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:48.252161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:48.252200 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:48.255135 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:48.255914 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:48.255920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:48.255925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:48.256070 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:48.256543 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:48.256547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:48.256549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:48.256861 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:48.257028 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:48.257825 (+ 797us) server_negotiation.cc:299] Negotiation successful
0807 11:33:48.258095 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:33:48.259907 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.240577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:34969)
0807 11:33:48.241360 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:48.241398 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:48.243215 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:48.243957 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:48.243965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:48.244460 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:33:48.245007 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.245023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.246070 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.246074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.247497 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.247517 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.248628 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.248635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.248967 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.249653 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:33:48.249681 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:33:48.251891 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:33:48.255293 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:48.255302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:48.255320 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:48.255770 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:48.256192 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:48.256198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:48.256203 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:48.256407 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:48.257001 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:33:48.257009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:33:48.257443 (+ 434us) client_negotiation.cc:769] Sending connection context
0807 11:33:48.257727 (+ 284us) client_negotiation.cc:240] Negotiation successful
0807 11:33:48.258301 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":311,"threads_started":1}
I20260807 11:33:48.261600 11251 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38339
I20260807 11:33:48.261932 11251 heartbeater.cc:464] Registering TS with master...
I20260807 11:33:48.262826 11251 heartbeater.cc:511] Master 127.24.103.126:38339 requested a full tablet report, sending...
I20260807 11:33:48.264698 10893 ts_manager.cc:194] Registered new tserver with Master: 72adee99138446e0b05e056656e36270 (127.24.103.66:34113)
I20260807 11:33:48.265802 10893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:34969
W20260807 11:33:48.604331 11254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:33:48.604728 11254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:33:48.604908 11254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:33:48.614012 11254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:33:48.614208 11254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:33:48.614398 11254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:33:48.624928 11254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:39911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=35107
--tserver_master_addrs=127.24.103.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:33:48.626348 11254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:33:48.627769 11254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:33:48.642705 11259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:48.642693 11260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:33:48.643278 11262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:33:48.643672 11254 server_base.cc:1034] running on GCE node
I20260807 11:33:48.644322 11254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:33:48.645452 11254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:33:48.646706 11254 hybrid_clock.cc:648] HybridClock initialized: now 1786102428646655 us; error 46 us; skew 500 ppm
I20260807 11:33:48.647125 11254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:33:48 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102428, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:33:48.659548 11254 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:33:48.661703 11254 webserver.cc:460] Webserver started at http://127.24.103.67:35107/ using document root <none> and password file <none>
I20260807 11:33:48.662611 11254 fs_manager.cc:359] Metadata directory not provided
I20260807 11:33:48.662786 11254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:33:48.668915 11254 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 11:33:48.672178 11270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:48.673854 11254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:33:48.674038 11254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "3ebf7c502b114b1785348f2ae12917f1"
format_stamp: "Formatted at 2026-08-07 11:33:25 on dist-test-slave-wl2h"
server_key: "96df2dd6e8eb857b6475c3311d42fb2c"
server_key_iv: "3983ce24c4a48aa97e7a1038683c3055"
server_key_version: "encryptionkey@0"
I20260807 11:33:48.675278 11254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:33:48.707952 11254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:33:48.712960 11254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:33:48.713284 11254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:33:48.714463 11254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:33:48.715941 11254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:33:48.716084 11254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:48.716224 11254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:33:48.716317 11254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:33:48.753849 11254 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:39911
I20260807 11:33:48.753964 11395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:39911 every 8 connection(s)
I20260807 11:33:48.755450 11254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:33:48.764909 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 11254
Aug 07 11:33:48 dist-test-slave-wl2h krb5kdc[10239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102428, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:33:48.778120 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.758158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:37587 (local address 127.24.103.126:38339)
0807 11:33:48.758412 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:48.758419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:48.759677 (+ 1258us) server_negotiation.cc:407] Connection header received
0807 11:33:48.761552 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:48.761556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:48.761625 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:48.761732 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:48.763286 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.764019 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.765948 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.766771 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.766914 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.770950 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:48.770956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:48.770961 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:48.771001 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:48.774554 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:48.775405 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:48.775412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:48.775416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:48.775574 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:48.776147 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:48.776151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:48.776153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:48.776438 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:48.776573 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:48.777523 (+ 950us) server_negotiation.cc:299] Negotiation successful
0807 11:33:48.777737 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 11:33:48.779567 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.758621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:37587)
0807 11:33:48.759489 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:48.759543 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:48.761286 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:48.761945 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:48.761953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:48.762467 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:33:48.763113 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.763142 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.764202 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.764207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.765737 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.765749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.767063 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.767073 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.767403 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.768167 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:33:48.768203 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:33:48.770727 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:33:48.774692 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:48.774699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:48.774710 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:48.775275 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:48.775727 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:33:48.775731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:33:48.775733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:33:48.775992 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:33:48.776579 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:33:48.776590 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:33:48.777024 (+ 434us) client_negotiation.cc:769] Sending connection context
0807 11:33:48.777401 (+ 377us) client_negotiation.cc:240] Negotiation successful
0807 11:33:48.777974 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":404,"threads_started":1}
I20260807 11:33:48.781203 11396 heartbeater.cc:347] Connected to a master server at 127.24.103.126:38339
I20260807 11:33:48.781545 11396 heartbeater.cc:464] Registering TS with master...
I20260807 11:33:48.782502 11396 heartbeater.cc:511] Master 127.24.103.126:38339 requested a full tablet report, sending...
I20260807 11:33:48.783277 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.765978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.24.103.126:38339)
0807 11:33:48.766846 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:48.766853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:48.766872 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:33:48.766948 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:48.766954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:48.767053 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:48.767161 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:48.769306 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.770045 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.771956 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.772446 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.772573 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.775966 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:48.775972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:48.775978 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:48.776020 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:48.779415 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:48.780485 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:48.780490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:48.780493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:48.780647 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:48.781546 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:48.781551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:48.781554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:48.781914 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:48.782209 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:48.782569 (+ 360us) server_negotiation.cc:299] Negotiation successful
0807 11:33:48.782810 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":618,"threads_started":1}
I20260807 11:33:48.784210 10892 ts_manager.cc:194] Registered new tserver with Master: 3ebf7c502b114b1785348f2ae12917f1 (127.24.103.67:39911)
I20260807 11:33:48.785341 10892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:37587
I20260807 11:33:48.786170 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:33:48.791721 11102 heartbeater.cc:503] Master 127.24.103.126:38339 was elected leader, sending a full tablet report...
I20260807 11:33:48.794032 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.787214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.24.103.126:38339)
0807 11:33:48.787433 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:48.787439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:48.787950 (+ 511us) server_negotiation.cc:407] Connection header received
0807 11:33:48.788117 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:48.788125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:48.788341 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:48.788484 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:33:48.789016 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.789754 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.791629 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.792250 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.792385 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.793011 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:33:48.793404 (+ 393us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 11:33:48.793822 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54496: token expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260807 11:33:48.794191 11403 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54496: token expired
W20260807 11:33:48.794818 10800 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.103.126:38339)
I20260807 11:33:48.808532 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.795368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.24.103.126:38339)
0807 11:33:48.795581 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:48.795587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:48.795710 (+ 123us) server_negotiation.cc:407] Connection header received
0807 11:33:48.795903 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:48.795908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:48.795981 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:48.796083 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:48.797490 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.798307 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.799871 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.800304 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.800440 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.803148 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:48.803161 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:48.803165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:48.803198 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:48.805747 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:48.806466 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:48.806473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:48.806476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:48.806621 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:48.807044 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:48.807049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:48.807052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:48.807366 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:48.807662 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:48.807943 (+ 281us) server_negotiation.cc:299] Negotiation successful
0807 11:33:48.808179 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 11:33:48.812677 10892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:33:48.820294 10892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:33:48.847393 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.832982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:39911 (local address 127.0.0.1:54680)
0807 11:33:48.835329 (+ 2347us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:48.835382 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:48.835610 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:48.839606 (+ 3996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:48.839619 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:48.839666 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:48.840355 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.840374 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.842439 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.842444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.844185 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.844198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.846303 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.846310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.846518 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.846583 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 11:33:48.846728 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 11:33:48.846935 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1932,"mutex_wait_us":1,"thread_start_us":6656,"threads_started":1}
I20260807 11:33:48.847438 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.833134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.24.103.67:39911)
0807 11:33:48.838600 (+ 5466us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:48.838612 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:48.838644 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:33:48.838726 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:48.838731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:48.838926 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:48.839205 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:48.840619 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.842293 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.844369 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.846051 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.846198 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.846442 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:48.846804 (+ 362us) server_negotiation.cc:299] Negotiation successful
0807 11:33:48.847009 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5306,"thread_start_us":5147,"threads_started":1}
I20260807 11:33:48.849002 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.833463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.24.103.65:42241)
0807 11:33:48.836424 (+ 2961us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:48.836437 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:48.840435 (+ 3998us) server_negotiation.cc:407] Connection header received
0807 11:33:48.840580 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:48.840589 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:48.840839 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:48.841051 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:48.841969 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.843789 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.846309 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.847872 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.848025 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.848255 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:48.848518 (+ 263us) server_negotiation.cc:299] Negotiation successful
0807 11:33:48.848681 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2713,"thread_start_us":3023,"threads_started":1}
I20260807 11:33:48.849165 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.833002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42241 (local address 127.0.0.1:37772)
0807 11:33:48.840044 (+ 7042us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:48.840070 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:48.840254 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:48.841321 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:48.841326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:48.841354 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:48.841767 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.841789 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.843945 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.843950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.846077 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.846090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.848117 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.848123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.848301 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.848337 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 11:33:48.848444 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 11:33:48.848576 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6895,"mutex_wait_us":131,"thread_start_us":1018,"threads_started":1}
I20260807 11:33:48.850004 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.835726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34113 (local address 127.0.0.1:56668)
0807 11:33:48.840547 (+ 4821us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:48.840569 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:48.840782 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:48.841654 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:48.841658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:48.841687 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:48.842128 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.842139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.844165 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.844171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.846650 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.846664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.849088 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.849095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.849309 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.849348 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 11:33:48.849463 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 11:33:48.849603 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4674,"mutex_wait_us":1,"thread_start_us":4498,"threads_started":1}
I20260807 11:33:48.850051 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.836290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.24.103.66:34113)
0807 11:33:48.839516 (+ 3226us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:48.839526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:48.840668 (+ 1142us) server_negotiation.cc:407] Connection header received
0807 11:33:48.840982 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:48.840989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:48.841213 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:48.841370 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:48.842320 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.844016 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.846876 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.848510 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.848643 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.848830 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:48.849530 (+ 700us) server_negotiation.cc:299] Negotiation successful
0807 11:33:48.849713 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3043,"thread_start_us":2925,"threads_started":1}
I20260807 11:33:48.852432 11325 tablet_service.cc:1467] Processing CreateTablet for tablet dcd62c26f1a74ed3b8d19949ee68d2bc (DEFAULT_TABLE table=test-table [id=1d557f0238604698a3faf1d0214b5ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:48.853278 11030 tablet_service.cc:1467] Processing CreateTablet for tablet dcd62c26f1a74ed3b8d19949ee68d2bc (DEFAULT_TABLE table=test-table [id=1d557f0238604698a3faf1d0214b5ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:48.853855 11325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd62c26f1a74ed3b8d19949ee68d2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:48.854756 11030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd62c26f1a74ed3b8d19949ee68d2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:48.854537 11180 tablet_service.cc:1467] Processing CreateTablet for tablet dcd62c26f1a74ed3b8d19949ee68d2bc (DEFAULT_TABLE table=test-table [id=1d557f0238604698a3faf1d0214b5ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:33:48.855863 11180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd62c26f1a74ed3b8d19949ee68d2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:33:48.868041 11419 tablet_bootstrap.cc:492] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1: Bootstrap starting.
I20260807 11:33:48.870589 11421 tablet_bootstrap.cc:492] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61: Bootstrap starting.
I20260807 11:33:48.872246 11422 tablet_bootstrap.cc:492] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: Bootstrap starting.
I20260807 11:33:48.872255 11419 tablet_bootstrap.cc:654] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:48.874590 11421 tablet_bootstrap.cc:654] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:48.874684 11419 log.cc:824] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:48.876030 11422 tablet_bootstrap.cc:654] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: Neither blocks nor log segments found. Creating new log.
I20260807 11:33:48.876048 11421 log.cc:824] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:48.877394 11422 log.cc:824] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: Log is configured to *not* fsync() on all Append() calls
I20260807 11:33:48.878482 11419 tablet_bootstrap.cc:492] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1: No bootstrap required, opened a new log
I20260807 11:33:48.878859 11419 ts_tablet_manager.cc:1397] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260807 11:33:48.879616 11421 tablet_bootstrap.cc:492] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61: No bootstrap required, opened a new log
I20260807 11:33:48.879901 11421 ts_tablet_manager.cc:1397] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 11:33:48.880254 11422 tablet_bootstrap.cc:492] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: No bootstrap required, opened a new log
I20260807 11:33:48.880514 11422 ts_tablet_manager.cc:1397] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 11:33:48.888224 11422 raft_consensus.cc:348] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } } peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } }
I20260807 11:33:48.888641 11422 raft_consensus.cc:374] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:48.888804 11422 raft_consensus.cc:729] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72adee99138446e0b05e056656e36270, State: Initialized, Role: FOLLOWER
I20260807 11:33:48.889424 11422 consensus_queue.cc:260] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } } peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } }
I20260807 11:33:48.890604 11251 heartbeater.cc:503] Master 127.24.103.126:38339 was elected leader, sending a full tablet report...
I20260807 11:33:48.891422 11422 ts_tablet_manager.cc:1428] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260807 11:33:48.891831 11421 raft_consensus.cc:348] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } } peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } }
I20260807 11:33:48.891831 11419 raft_consensus.cc:348] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } } peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } }
I20260807 11:33:48.892385 11421 raft_consensus.cc:374] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:48.892383 11419 raft_consensus.cc:374] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:33:48.892508 11421 raft_consensus.cc:729] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3e3da6e73154e4888a9a710f4c58f61, State: Initialized, Role: FOLLOWER
I20260807 11:33:48.893208 11421 consensus_queue.cc:260] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } } peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } }
I20260807 11:33:48.895067 11419 raft_consensus.cc:729] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ebf7c502b114b1785348f2ae12917f1, State: Initialized, Role: FOLLOWER
I20260807 11:33:48.895253 11421 ts_tablet_manager.cc:1428] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260807 11:33:48.896034 11419 consensus_queue.cc:260] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } } peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } }
I20260807 11:33:48.898490 11396 heartbeater.cc:503] Master 127.24.103.126:38339 was elected leader, sending a full tablet report...
I20260807 11:33:48.899423 11419 ts_tablet_manager.cc:1428] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1: Time spent starting tablet: real 0.020s user 0.013s sys 0.003s
I20260807 11:33:48.916359 11428 raft_consensus.cc:482] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:33:48.916770 11428 raft_consensus.cc:504] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } } peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } }
I20260807 11:33:48.918516 11428 leader_election.cc:288] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3e3da6e73154e4888a9a710f4c58f61 (127.24.103.65:42241), 72adee99138446e0b05e056656e36270 (127.24.103.66:34113)
I20260807 11:33:48.924602 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.918781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:42241 (local address 127.24.103.67:37431)
0807 11:33:48.919069 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:48.919093 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:48.919244 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:48.919876 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:48.919881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:48.919918 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:48.920083 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.920094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.921226 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.921234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.922658 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.922670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.923846 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.923852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.924045 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.924070 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 11:33:48.924184 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 11:33:48.924318 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260807 11:33:48.924602 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.919079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:37431 (local address 127.24.103.65:42241)
0807 11:33:48.919317 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:48.919324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:48.919345 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:33:48.919423 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:48.919428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:48.919518 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:48.919648 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:48.920270 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.921086 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.922830 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.923681 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.923862 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.923940 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:48.924213 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 11:33:48.924344 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 11:33:48.925792 11052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd62c26f1a74ed3b8d19949ee68d2bc" candidate_uuid: "3ebf7c502b114b1785348f2ae12917f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e3da6e73154e4888a9a710f4c58f61" is_pre_election: true
I20260807 11:33:48.926272 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.919731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:51513 (local address 127.24.103.66:34113)
0807 11:33:48.919985 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:48.919991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:48.920224 (+ 233us) server_negotiation.cc:407] Connection header received
0807 11:33:48.920456 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:48.920462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:48.920539 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:48.920648 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:33:48.921283 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.922113 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.924052 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.925268 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.925456 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.925536 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:48.925834 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 11:33:48.925978 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260807 11:33:48.926307 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.919381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:34113 (local address 127.24.103.67:51513)
0807 11:33:48.920104 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0807 11:33:48.920124 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:33:48.920301 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:33:48.920696 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:33:48.920704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:33:48.920730 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:33:48.921126 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.921137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.922275 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.922286 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.923874 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:33:48.923886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.925460 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.925466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:33:48.925652 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.925684 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 11:33:48.925804 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 11:33:48.925941 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":347,"threads_started":1}
I20260807 11:33:48.926420 11052 raft_consensus.cc:2454] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ebf7c502b114b1785348f2ae12917f1 in term 0.
I20260807 11:33:48.927197 11271 leader_election.cc:302] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ebf7c502b114b1785348f2ae12917f1, e3e3da6e73154e4888a9a710f4c58f61; no voters:
I20260807 11:33:48.927289 11203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd62c26f1a74ed3b8d19949ee68d2bc" candidate_uuid: "3ebf7c502b114b1785348f2ae12917f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72adee99138446e0b05e056656e36270" is_pre_election: true
I20260807 11:33:48.927603 11428 raft_consensus.cc:2781] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:33:48.927778 11428 raft_consensus.cc:482] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:33:48.927884 11428 raft_consensus.cc:3037] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:48.927826 11203 raft_consensus.cc:2454] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ebf7c502b114b1785348f2ae12917f1 in term 0.
I20260807 11:33:48.930467 11428 raft_consensus.cc:504] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } } peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } }
I20260807 11:33:48.931313 11428 leader_election.cc:288] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [CANDIDATE]: Term 1 election: Requested vote from peers e3e3da6e73154e4888a9a710f4c58f61 (127.24.103.65:42241), 72adee99138446e0b05e056656e36270 (127.24.103.66:34113)
I20260807 11:33:48.931692 11052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd62c26f1a74ed3b8d19949ee68d2bc" candidate_uuid: "3ebf7c502b114b1785348f2ae12917f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e3da6e73154e4888a9a710f4c58f61"
I20260807 11:33:48.931805 11203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd62c26f1a74ed3b8d19949ee68d2bc" candidate_uuid: "3ebf7c502b114b1785348f2ae12917f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72adee99138446e0b05e056656e36270"
I20260807 11:33:48.931917 11052 raft_consensus.cc:3037] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:48.932082 11203 raft_consensus.cc:3037] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:33:48.934397 11203 raft_consensus.cc:2454] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ebf7c502b114b1785348f2ae12917f1 in term 1.
I20260807 11:33:48.934428 11052 raft_consensus.cc:2454] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ebf7c502b114b1785348f2ae12917f1 in term 1.
I20260807 11:33:48.934845 11271 leader_election.cc:302] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ebf7c502b114b1785348f2ae12917f1, 72adee99138446e0b05e056656e36270; no voters:
I20260807 11:33:48.935210 11428 raft_consensus.cc:2781] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:33:48.935750 11428 raft_consensus.cc:686] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 1 LEADER]: Becoming Leader. State: Replica: 3ebf7c502b114b1785348f2ae12917f1, State: Running, Role: LEADER
I20260807 11:33:48.936360 11428 consensus_queue.cc:237] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } } peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } }
I20260807 11:33:48.941875 10892 catalog_manager.cc:5495] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ebf7c502b114b1785348f2ae12917f1 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "3ebf7c502b114b1785348f2ae12917f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ebf7c502b114b1785348f2ae12917f1" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 39911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 } health_report { overall_health: UNKNOWN } } }
I20260807 11:33:48.974852 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:48.968658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54692 (local address 127.24.103.67:39911)
0807 11:33:48.968864 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:48.968869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:48.968989 (+ 120us) server_negotiation.cc:407] Connection header received
0807 11:33:48.969180 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:48.969184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:48.969284 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:48.969413 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:33:48.969871 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.970693 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.972383 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:48.972821 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:48.972940 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:48.973603 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:33:48.974204 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:33:48.974327 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:48.974541 (+ 214us) server_negotiation.cc:299] Negotiation successful
0807 11:33:48.974624 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 11:33:48.984421 11052 raft_consensus.cc:1261] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61 [term 1 FOLLOWER]: Refusing update from remote peer 3ebf7c502b114b1785348f2ae12917f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:33:48.984612 11203 raft_consensus.cc:1261] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270 [term 1 FOLLOWER]: Refusing update from remote peer 3ebf7c502b114b1785348f2ae12917f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:33:48.985597 11428 consensus_queue.cc:1035] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3e3da6e73154e4888a9a710f4c58f61" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 42241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:33:48.986019 11430 consensus_queue.cc:1035] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72adee99138446e0b05e056656e36270" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 34113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 11:33:48.989524 11252 tablet.cc:2369] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:33:49.000340 11434 mvcc.cc:205] Tried to move back new op lower bound from 7315875549108793344 to 7315875548928004096. Current Snapshot: MvccSnapshot[applied={T|T < 7315875549108793344}]
I20260807 11:33:49.000669 11435 mvcc.cc:205] Tried to move back new op lower bound from 7315875549108793344 to 7315875548928004096. Current Snapshot: MvccSnapshot[applied={T|T < 7315875549108793344}]
I20260807 11:33:49.000645 11436 mvcc.cc:205] Tried to move back new op lower bound from 7315875549108793344 to 7315875548928004096. Current Snapshot: MvccSnapshot[applied={T|T < 7315875549108793344}]
I20260807 11:33:49.016714 10893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:33:49.017010 10893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:33:49.021190 10893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Sending DeleteTablet for 3 replicas of tablet dcd62c26f1a74ed3b8d19949ee68d2bc
I20260807 11:33:49.022667 11325 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd62c26f1a74ed3b8d19949ee68d2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:49 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:54680
I20260807 11:33:49.022727 11030 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd62c26f1a74ed3b8d19949ee68d2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:49 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:37772
I20260807 11:33:49.022974 11180 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd62c26f1a74ed3b8d19949ee68d2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:33:49 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:56668
I20260807 11:33:49.023643 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 10934
I20260807 11:33:49.033676 11446 tablet_replica.cc:330] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: stopping tablet replica
I20260807 11:33:49.035235 11446 raft_consensus.cc:2229] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:33:49.035413 11445 tablet_replica.cc:330] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61: stopping tablet replica
I20260807 11:33:49.036010 11445 raft_consensus.cc:2229] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:33:49.036644 11445 raft_consensus.cc:2258] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:49.036648 11446 raft_consensus.cc:2258] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:49.037609 11443 tablet_replica.cc:330] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1: stopping tablet replica
I20260807 11:33:49.038801 11443 raft_consensus.cc:2229] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:33:49.039638 11443 raft_consensus.cc:2258] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:33:49.039916 11446 ts_tablet_manager.cc:1905] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:49.040274 11445 ts_tablet_manager.cc:1905] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:49.045354 11443 ts_tablet_manager.cc:1905] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:33:49.047510 11446 ts_tablet_manager.cc:1918] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:49.047746 11446 log.cc:1196] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/dcd62c26f1a74ed3b8d19949ee68d2bc
I20260807 11:33:49.048391 11446 ts_tablet_manager.cc:1939] T dcd62c26f1a74ed3b8d19949ee68d2bc P 72adee99138446e0b05e056656e36270: Deleting consensus metadata
I20260807 11:33:49.049950 11445 ts_tablet_manager.cc:1918] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:49.050242 11445 log.cc:1196] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/dcd62c26f1a74ed3b8d19949ee68d2bc
I20260807 11:33:49.050529 10875 catalog_manager.cc:4841] TS 72adee99138446e0b05e056656e36270 (127.24.103.66:34113): tablet dcd62c26f1a74ed3b8d19949ee68d2bc (table test-table [id=1d557f0238604698a3faf1d0214b5ce9]) successfully deleted
I20260807 11:33:49.051090 11445 ts_tablet_manager.cc:1939] T dcd62c26f1a74ed3b8d19949ee68d2bc P e3e3da6e73154e4888a9a710f4c58f61: Deleting consensus metadata
I20260807 11:33:49.053004 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:49.024726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.24.103.65:42241)
0807 11:33:49.025391 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:49.025398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:49.025419 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:33:49.025485 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:49.025492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:49.025587 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:49.025710 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:49.028118 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:49.029522 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:49.031869 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:49.032632 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:49.032794 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:49.040166 (+ 7372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:49.040180 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:49.040197 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:49.040676 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:49.048107 (+ 7431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:49.049274 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:49.049283 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:49.049287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:49.049452 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:49.049952 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:49.049957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:49.049960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:49.050509 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:49.051302 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:49.052332 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0807 11:33:49.052589 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512}
I20260807 11:33:49.053776 11096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:49.053740 10875 catalog_manager.cc:4841] TS e3e3da6e73154e4888a9a710f4c58f61 (127.24.103.65:42241): tablet dcd62c26f1a74ed3b8d19949ee68d2bc (table test-table [id=1d557f0238604698a3faf1d0214b5ce9]) successfully deleted
I20260807 11:33:49.058585 11443 ts_tablet_manager.cc:1918] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:33:49.058802 11443 log.cc:1196] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/dcd62c26f1a74ed3b8d19949ee68d2bc
I20260807 11:33:49.059365 11443 ts_tablet_manager.cc:1939] T dcd62c26f1a74ed3b8d19949ee68d2bc P 3ebf7c502b114b1785348f2ae12917f1: Deleting consensus metadata
I20260807 11:33:49.060956 10877 catalog_manager.cc:4841] TS 3ebf7c502b114b1785348f2ae12917f1 (127.24.103.67:39911): tablet dcd62c26f1a74ed3b8d19949ee68d2bc (table test-table [id=1d557f0238604698a3faf1d0214b5ce9]) successfully deleted
I20260807 11:33:49.164307 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 10934
I20260807 11:33:49.188884 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 11106
I20260807 11:33:49.203728 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:49.189760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.24.103.66:34113)
0807 11:33:49.189980 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:49.189986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:49.190007 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:33:49.190121 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:49.190126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:49.190205 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:49.190314 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:49.191785 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:49.192544 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:49.193853 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:49.194260 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:49.194399 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:49.197252 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:49.197268 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:49.197283 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:49.197331 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:49.200514 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:49.201465 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:49.201473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:49.201475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:49.201632 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:49.202087 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:49.202120 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:49.202122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:49.202504 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:49.202733 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:49.203243 (+ 510us) server_negotiation.cc:299] Negotiation successful
0807 11:33:49.203411 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:33:49.204480 11246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:49.304075 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 11106
I20260807 11:33:49.326107 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 11254
I20260807 11:33:49.342374 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:33:49.326999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54694 (local address 127.24.103.67:39911)
0807 11:33:49.327253 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 11:33:49.327259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:33:49.327334 (+ 75us) server_negotiation.cc:407] Connection header received
0807 11:33:49.327534 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:33:49.327568 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:33:49.327669 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:33:49.327786 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:33:49.329182 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:49.329985 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:49.331476 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:33:49.331954 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:33:49.332142 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:33:49.335128 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:33:49.335140 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:33:49.335156 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:33:49.335210 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:33:49.338727 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:49.339882 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:49.339890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:49.339893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:49.340058 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:33:49.340519 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:33:49.340529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:33:49.340531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:33:49.340905 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:33:49.341316 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0807 11:33:49.341693 (+ 377us) server_negotiation.cc:299] Negotiation successful
0807 11:33:49.341941 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260807 11:33:49.343360 11390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:49.476004 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 11254
I20260807 11:33:49.501392 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 10855
I20260807 11:33:49.502632 10927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:33:49.620239 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 10855
I20260807 11:34:09.648043 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38339
--webserver_interface=127.24.103.126
--webserver_port=39071
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:10.007331 11451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:10.007881 11451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:10.008103 11451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:10.017196 11451 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 11:34:10.017385 11451 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 11:34:10.017473 11451 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 11:34:10.017576 11451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:34:10.017678 11451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:10.017794 11451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:10.017897 11451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:34:10.017990 11451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:34:10.029527 11451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=39071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:10.031055 11451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:10.032615 11451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:10.046094 11457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:10.046191 11456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:10.046068 11459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:10.051909 11451 server_base.cc:1034] running on GCE node
I20260807 11:34:10.052891 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:10.054445 11451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:10.055724 11451 hybrid_clock.cc:648] HybridClock initialized: now 1786102450055634 us; error 62 us; skew 500 ppm
I20260807 11:34:10.056149 11451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:10 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102450, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:10.064339 11451 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:34:10.066248 11451 webserver.cc:460] Webserver started at http://127.24.103.126:39071/ using document root <none> and password file <none>
I20260807 11:34:10.067121 11451 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:10.067279 11451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:10.073311 11451 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:34:10.076581 11466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:10.078117 11451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:34:10.078333 11451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "29a995ac34994ee08a6e36281f2aa3a4"
format_stamp: "Formatted at 2026-08-07 11:33:23 on dist-test-slave-wl2h"
server_key: "467439eb0947f2c02311d3c16ab8673c"
server_key_iv: "3df0753c2ced1a57e9518b477aab710b"
server_key_version: "encryptionkey@0"
I20260807 11:34:10.079473 11451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:10.097405 11451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:10.101919 11451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:10.102228 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:10.123286 11451 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38339
I20260807 11:34:10.123323 11525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38339 every 8 connection(s)
I20260807 11:34:10.124879 11451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:34:10.131794 11526 sys_catalog.cc:263] Verifying existing consensus state
I20260807 11:34:10.132824 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 11451
I20260807 11:34:10.133739 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:42241
--local_ip_for_outbound_sockets=127.24.103.65
--tserver_master_addrs=127.24.103.126:38339
--webserver_port=43597
--webserver_interface=127.24.103.65
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:34:10.135259 11526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Bootstrap starting.
I20260807 11:34:10.155753 11526 log.cc:824] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:10.171958 11526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 11:34:10.172585 11526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Bootstrap complete.
I20260807 11:34:10.181994 11526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:34:10.182933 11526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a995ac34994ee08a6e36281f2aa3a4, State: Initialized, Role: FOLLOWER
I20260807 11:34:10.183722 11526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:34:10.184006 11526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:34:10.184136 11526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:34:10.184316 11526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 2 FOLLOWER]: Advancing to term 3
I20260807 11:34:10.188376 11526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:34:10.188892 11526 leader_election.cc:302] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29a995ac34994ee08a6e36281f2aa3a4; no voters:
I20260807 11:34:10.189604 11526 leader_election.cc:288] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 11:34:10.189741 11531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 11:34:10.191154 11531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [term 3 LEADER]: Becoming Leader. State: Replica: 29a995ac34994ee08a6e36281f2aa3a4, State: Running, Role: LEADER
I20260807 11:34:10.191788 11531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } }
I20260807 11:34:10.191972 11526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:34:10.194483 11534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29a995ac34994ee08a6e36281f2aa3a4. Latest consensus state: current_term: 3 leader_uuid: "29a995ac34994ee08a6e36281f2aa3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } } }
I20260807 11:34:10.194902 11534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:10.195541 11533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "29a995ac34994ee08a6e36281f2aa3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a995ac34994ee08a6e36281f2aa3a4" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38339 } } }
I20260807 11:34:10.195865 11533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:10.205754 11539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:34:10.215483 11539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:34:10.217278 11539 catalog_manager.cc:1259] Loaded cluster ID: a0d465aa9eaa4cac946a486e92385a2c
I20260807 11:34:10.217460 11539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:34:10.222064 11539 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:34:10.223759 11539 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Loaded TSK: 0
I20260807 11:34:10.223989 11539 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29a995ac34994ee08a6e36281f2aa3a4: Loaded TSK: 1
F20260807 11:34:10.225168 11539 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f396fd039ad google::LogMessage::Fail() at ??:0
@ 0x7f396fd0688a google::LogMessage::SendToLog() at ??:0
@ 0x7f396fd034de google::LogMessage::Flush() at ??:0
@ 0x7f396fd04a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3973a677d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3973a5673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f39857321c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3985731783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f39857bd2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f39857bd0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f39858075e9 std::function<>::operator()() at ??:0
@ 0x7f3985735cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3985733c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f39857bb610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f39857bb439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f397294f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f397295291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3972952799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f397292deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3975c076db start_thread at ??:0
@ 0x7f396eb9d61f clone at ??:0
W20260807 11:34:10.535351 11529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:10.535701 11529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:10.535882 11529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:10.543318 11529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:10.543486 11529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:10.543684 11529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:34:10.558425 11529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:42241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=43597
--tserver_master_addrs=127.24.103.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:10.559934 11529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:10.561323 11529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:10.577591 11583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:10.578289 11580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:10.579080 11579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:10.580989 11529 server_base.cc:1034] running on GCE node
I20260807 11:34:10.581780 11529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:10.583127 11529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:10.584400 11529 hybrid_clock.cc:648] HybridClock initialized: now 1786102450584338 us; error 57 us; skew 500 ppm
I20260807 11:34:10.584827 11529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:10 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102450, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:10.588711 11529 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:34:10.590469 11529 webserver.cc:460] Webserver started at http://127.24.103.65:43597/ using document root <none> and password file <none>
I20260807 11:34:10.591362 11529 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:10.591516 11529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:10.597740 11529 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 11:34:10.600979 11590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:10.602460 11529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:34:10.602667 11529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "e3e3da6e73154e4888a9a710f4c58f61"
format_stamp: "Formatted at 2026-08-07 11:33:24 on dist-test-slave-wl2h"
server_key: "328e255a285a9a94c796c45c91d606cc"
server_key_iv: "e0a1c0e9cd28e8dd08691bda087c210b"
server_key_version: "encryptionkey@0"
I20260807 11:34:10.604139 11529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:10.630339 11529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:10.635553 11529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:10.635861 11529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:10.636971 11529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:10.638424 11529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:10.638554 11529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:10.638701 11529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:10.638795 11529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:10.676883 11529 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:42241
I20260807 11:34:10.676944 11718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:42241 every 8 connection(s)
I20260807 11:34:10.678395 11529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:34:10.683080 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 11529
I20260807 11:34:10.683770 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:34113
--local_ip_for_outbound_sockets=127.24.103.66
--tserver_master_addrs=127.24.103.126:38339
--webserver_port=39641
--webserver_interface=127.24.103.66
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:34:10.684262 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:10.681576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:34905)
0807 11:34:10.682568 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:10.682934 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":336,"threads_started":1}
W20260807 11:34:10.685083 11719 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:38339 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:38339: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
I20260807 11:34:10.686133 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:10.685661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:54733)
0807 11:34:10.685879 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:10.685980 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 11:34:10.687173 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:10.686667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:40205)
0807 11:34:10.686860 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:10.686961 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 11:34:10.687460 11719 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 11:34:11.041307 11722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:11.041702 11722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:11.041865 11722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:11.049852 11722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:11.050101 11722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:11.050348 11722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:34:11.061362 11722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:34113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=39641
--tserver_master_addrs=127.24.103.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:11.062811 11722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:11.064225 11722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:11.077907 11729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:11.078096 11730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:11.078429 11732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:11.079360 11722 server_base.cc:1034] running on GCE node
I20260807 11:34:11.080034 11722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:11.081118 11722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:11.082345 11722 hybrid_clock.cc:648] HybridClock initialized: now 1786102451082271 us; error 72 us; skew 500 ppm
I20260807 11:34:11.082695 11722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:11 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102451, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:11.086054 11722 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:34:11.087864 11722 webserver.cc:460] Webserver started at http://127.24.103.66:39641/ using document root <none> and password file <none>
I20260807 11:34:11.088626 11722 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:11.088791 11722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:11.094594 11722 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 11:34:11.097988 11739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:11.099628 11722 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:34:11.099828 11722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "72adee99138446e0b05e056656e36270"
format_stamp: "Formatted at 2026-08-07 11:33:24 on dist-test-slave-wl2h"
server_key: "ae50515a9b92654c6e2f6535a4402cb5"
server_key_iv: "19b5aac1d14793fe212f29657a6d5d80"
server_key_version: "encryptionkey@0"
I20260807 11:34:11.100957 11722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:11.135587 11722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:11.141651 11722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:11.141971 11722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:11.143163 11722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:11.144793 11722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:11.144927 11722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:11.145069 11722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:11.145172 11722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:11.182463 11722 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:34113
I20260807 11:34:11.182516 11868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:34113 every 8 connection(s)
I20260807 11:34:11.183975 11722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:34:11.187701 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 11722
I20260807 11:34:11.188354 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:39911
--local_ip_for_outbound_sockets=127.24.103.67
--tserver_master_addrs=127.24.103.126:38339
--webserver_port=35107
--webserver_interface=127.24.103.67
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:34:11.189471 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:11.187173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:36905)
0807 11:34:11.188125 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:11.188485 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":403,"threads_started":1}
W20260807 11:34:11.190301 11869 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:38339 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:38339: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
I20260807 11:34:11.191510 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:11.191103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:40187)
0807 11:34:11.191319 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:11.191391 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 11:34:11.192600 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:11.192112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:48493)
0807 11:34:11.192310 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:11.192428 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 11:34:11.192907 11869 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 11:34:11.541057 11872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:11.541356 11872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:11.541538 11872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:11.548887 11872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:11.549000 11872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:11.549208 11872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:34:11.559424 11872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:39911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=35107
--tserver_master_addrs=127.24.103.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:11.560794 11872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:11.562144 11872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:11.576639 11882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:11.576684 11879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:11.576906 11878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:11.577652 11872 server_base.cc:1034] running on GCE node
I20260807 11:34:11.578234 11872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:11.579270 11872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:11.580492 11872 hybrid_clock.cc:648] HybridClock initialized: now 1786102451580424 us; error 56 us; skew 500 ppm
I20260807 11:34:11.580806 11872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:11 dist-test-slave-wl2h krb5kdc[10239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102451, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:11.584247 11872 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:34:11.586054 11872 webserver.cc:460] Webserver started at http://127.24.103.67:35107/ using document root <none> and password file <none>
I20260807 11:34:11.586791 11872 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:11.586948 11872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:11.592665 11872 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 11:34:11.595485 11890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:11.596854 11872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 11:34:11.597054 11872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "3ebf7c502b114b1785348f2ae12917f1"
format_stamp: "Formatted at 2026-08-07 11:33:25 on dist-test-slave-wl2h"
server_key: "96df2dd6e8eb857b6475c3311d42fb2c"
server_key_iv: "3983ce24c4a48aa97e7a1038683c3055"
server_key_version: "encryptionkey@0"
I20260807 11:34:11.598028 11872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:11.622260 11872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:11.626686 11872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:11.627053 11872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:11.628196 11872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:11.629639 11872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:11.629784 11872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:11.629909 11872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:11.629998 11872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:11.665759 11872 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:39911
I20260807 11:34:11.665854 12012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:39911 every 8 connection(s)
I20260807 11:34:11.667340 11872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:34:11.671945 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 11872
I20260807 11:34:11.672520 12015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:11.670347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:43885)
0807 11:34:11.671224 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:11.671518 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":300,"threads_started":1}
W20260807 11:34:11.673354 12013 heartbeater.cc:644] Failed to heartbeat to 127.24.103.126:38339 (0 consecutive failures): Network error: Failed to ping master at 127.24.103.126:38339: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
I20260807 11:34:11.674486 12015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:11.674083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:37815)
0807 11:34:11.674266 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:11.674348 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 11:34:11.675573 12015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:11.675162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:49149)
0807 11:34:11.675317 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:11.675419 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 11:34:11.675848 12013 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 11:34:11.688910 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:11.688020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:43229)
0807 11:34:11.688513 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:11.688660 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":252,"threads_started":1}
I20260807 11:34:12.194401 12020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:12.193547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:46963)
0807 11:34:12.194038 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:12.194157 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":240,"threads_started":1}
I20260807 11:34:12.677363 12021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:12.676470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:51827)
0807 11:34:12.677027 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:12.677120 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":302,"threads_started":1}
I20260807 11:34:12.690811 12022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:12.689927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:43901)
0807 11:34:12.690542 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:12.690636 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":361,"threads_started":1}
I20260807 11:34:13.196430 12024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:13.195570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:46239)
0807 11:34:13.196145 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:13.196243 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":256,"threads_started":1}
I20260807 11:34:13.679160 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:13.678326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:42865)
0807 11:34:13.678826 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:13.678913 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":255,"threads_started":1}
I20260807 11:34:13.692811 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:13.691865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:49765)
0807 11:34:13.692514 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:13.692611 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":427,"threads_started":1}
I20260807 11:34:14.198315 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:14.197564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:40419)
0807 11:34:14.198043 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:14.198133 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":251,"threads_started":1}
I20260807 11:34:14.680881 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:14.680136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:38287)
0807 11:34:14.680595 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:14.680690 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260807 11:34:14.694679 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:14.693883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:56867)
0807 11:34:14.694413 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:14.694498 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":241,"threads_started":1}
I20260807 11:34:15.200073 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:15.199329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:54381)
0807 11:34:15.199800 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:15.199885 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260807 11:34:15.682622 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:15.681784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:35691)
0807 11:34:15.682365 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:15.682451 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":292,"threads_started":1}
I20260807 11:34:15.696559 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:15.695767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:42303)
0807 11:34:15.696279 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:15.696366 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":293,"threads_started":1}
I20260807 11:34:16.201959 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:16.201110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:57141)
0807 11:34:16.201695 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:16.201783 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":334,"threads_started":1}
I20260807 11:34:16.684716 12034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:16.683747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:46245)
0807 11:34:16.684362 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:16.684495 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":339,"threads_started":1}
I20260807 11:34:16.698349 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:16.697517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:46573)
0807 11:34:16.698081 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:16.698177 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":327,"threads_started":1}
I20260807 11:34:17.203954 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:17.203131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:45665)
0807 11:34:17.203663 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:17.203774 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":270,"threads_started":1}
I20260807 11:34:17.686657 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:17.685786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:40837)
0807 11:34:17.686358 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:17.686460 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":353,"threads_started":1}
I20260807 11:34:17.700318 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:17.699437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:42039)
0807 11:34:17.699964 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:17.700050 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":218,"threads_started":1}
I20260807 11:34:18.205961 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:18.204972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:54595)
0807 11:34:18.205630 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:18.205743 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":324,"threads_started":1}
I20260807 11:34:18.688625 12042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:18.687718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:45421)
0807 11:34:18.688304 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:18.688401 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":266,"threads_started":1}
I20260807 11:34:18.702423 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:18.701551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:57869)
0807 11:34:18.702047 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:18.702133 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":257,"threads_started":1}
I20260807 11:34:19.207737 12044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:19.206950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:54297)
0807 11:34:19.207470 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:19.207556 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":295,"threads_started":1}
I20260807 11:34:19.690876 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:19.689730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:36737)
0807 11:34:19.690483 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:19.690623 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":371,"threads_started":1}
I20260807 11:34:19.704434 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:19.703492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:55931)
0807 11:34:19.704042 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:19.704186 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":277,"threads_started":1}
I20260807 11:34:20.209827 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:20.208929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:57677)
0807 11:34:20.209527 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:20.209652 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":340,"threads_started":1}
I20260807 11:34:20.693030 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:20.692233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:35643)
0807 11:34:20.692743 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:20.692839 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":249,"threads_started":1}
I20260807 11:34:20.706408 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:20.705587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:52249)
0807 11:34:20.706135 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:20.706221 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
I20260807 11:34:21.211620 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:21.210843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:54913)
0807 11:34:21.211370 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:21.211451 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":322,"threads_started":1}
I20260807 11:34:21.695225 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:21.694175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:38573)
0807 11:34:21.694866 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:21.694972 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":420,"threads_started":1}
I20260807 11:34:21.708320 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:21.707470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:56143)
0807 11:34:21.707977 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:21.708088 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":264,"threads_started":1}
I20260807 11:34:22.213416 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:22.212634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:46069)
0807 11:34:22.213109 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:22.213216 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":228,"threads_started":1}
I20260807 11:34:22.697048 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:22.696252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:40077)
0807 11:34:22.696734 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:22.696843 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":255,"threads_started":1}
I20260807 11:34:22.710278 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:22.709419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:46451)
0807 11:34:22.709979 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:22.710063 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":290,"threads_started":1}
I20260807 11:34:23.215323 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:23.214483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:47793)
0807 11:34:23.214960 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:23.215156 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":254,"threads_started":1}
I20260807 11:34:23.698870 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:23.698118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:33587)
0807 11:34:23.698614 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:23.698709 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
I20260807 11:34:23.712337 12059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:23.711306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:38881)
0807 11:34:23.711975 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:23.712089 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":328,"threads_started":1}
I20260807 11:34:24.217170 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:24.216360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:38411)
0807 11:34:24.216863 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:24.216963 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":246,"threads_started":1}
I20260807 11:34:24.700804 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:24.699891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:42525)
0807 11:34:24.700461 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:24.700571 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":299,"threads_started":1}
I20260807 11:34:24.714183 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:24.713322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:47671)
0807 11:34:24.713834 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:24.713935 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":260,"threads_started":1}
I20260807 11:34:25.219118 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:25.218254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:55329)
0807 11:34:25.218750 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:25.218846 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":238,"threads_started":1}
I20260807 11:34:25.702550 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:25.701806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:39631)
0807 11:34:25.702274 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:25.702365 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":228,"threads_started":1}
I20260807 11:34:25.716331 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:25.715543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.65:33995)
0807 11:34:25.716014 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:25.716145 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":240,"threads_started":1}
I20260807 11:34:26.221379 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:26.220396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.66:38577)
0807 11:34:26.220936 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:26.221072 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":252,"threads_started":1}
I20260807 11:34:26.684141 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 11529
I20260807 11:34:26.705060 12067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:26.685413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.24.103.65:42241)
0807 11:34:26.686206 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:26.686228 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:26.686281 (+ 53us) server_negotiation.cc:407] Connection header received
0807 11:34:26.688221 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:26.688230 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:26.688690 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:26.689054 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:26.690786 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:26.692128 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:26.693513 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:26.694321 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:26.694526 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:26.697891 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:26.697907 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:26.697926 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:26.697978 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:26.701240 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:26.702031 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:26.702041 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:26.702057 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:26.702232 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:26.702643 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:26.702648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:26.702651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:26.703360 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:26.703665 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:26.704192 (+ 527us) server_negotiation.cc:299] Negotiation successful
0807 11:34:26.704643 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":375,"threads_started":1}
I20260807 11:34:26.705294 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:26.703857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38339 (local address 127.24.103.67:60213)
0807 11:34:26.704924 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:26.705088 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.126:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":818,"threads_started":1}
I20260807 11:34:26.706732 11714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:26.792865 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 11529
I20260807 11:34:26.816004 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 11722
I20260807 11:34:26.836246 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:26.817082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.24.103.66:34113)
0807 11:34:26.817687 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:26.817701 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:26.817754 (+ 53us) server_negotiation.cc:407] Connection header received
0807 11:34:26.819610 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:26.819620 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:26.820059 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:26.820451 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:26.822063 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:26.823672 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:26.825062 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:26.825776 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:26.826000 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:26.829000 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:26.829009 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:26.829024 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:26.829067 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:26.832386 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:26.833226 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:26.833234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:26.833240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:26.833434 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:26.833869 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:26.833875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:26.833881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:26.834455 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:26.834837 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:26.835449 (+ 612us) server_negotiation.cc:299] Negotiation successful
0807 11:34:26.835915 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":319,"threads_started":1}
I20260807 11:34:26.837688 11863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:26.924482 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 11722
I20260807 11:34:26.946550 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 11872
I20260807 11:34:26.965969 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:26.947727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.24.103.67:39911)
0807 11:34:26.948376 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:26.948392 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:26.948438 (+ 46us) server_negotiation.cc:407] Connection header received
0807 11:34:26.950124 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:26.950135 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:26.950549 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:26.950863 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:26.952575 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:26.953853 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:26.955213 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:26.955969 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:26.956128 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:26.958936 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:26.958945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:26.958961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:26.959001 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:26.962358 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:26.963232 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:26.963240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:26.963247 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:26.963439 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:26.963866 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:26.963870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:26.963872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:26.964459 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:26.964746 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:26.965138 (+ 392us) server_negotiation.cc:299] Negotiation successful
0807 11:34:26.965634 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":305,"threads_started":1}
I20260807 11:34:26.967648 12008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:27.054065 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 11872
2026-08-07T11:34:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65313 ms)
[----------] 30 tests from SecurityITest (313796 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:34:27 dist-test-slave-wl2h krb5kdc[12080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:34:27 dist-test-slave-wl2h krb5kdc[12080](info): set up 2 sockets
Aug 07 11:34:27 dist-test-slave-wl2h krb5kdc[12080](info): commencing operation
krb5kdc: starting...
W20260807 11:34:28.284962 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.007s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:34:28 dist-test-slave-wl2h krb5kdc[12080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:34:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:34:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:34:28.632618 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41533
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:41533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:29.010960 12095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:29.011463 12095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:29.011677 12095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:29.020262 12095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:34:29.020468 12095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:29.020597 12095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:29.020706 12095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:34:29.020804 12095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:34:29.031247 12095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:41533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:29.032795 12095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:29.034170 12095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:29.047235 12100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:29.047328 12101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:29.047273 12103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:29.048190 12095 server_base.cc:1034] running on GCE node
I20260807 11:34:29.048980 12095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:29.050411 12095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:29.051638 12095 hybrid_clock.cc:648] HybridClock initialized: now 1786102469051591 us; error 29 us; skew 500 ppm
I20260807 11:34:29.051977 12095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:29 dist-test-slave-wl2h krb5kdc[12080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102469, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:29.061517 12095 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:34:29.063700 12095 webserver.cc:460] Webserver started at http://127.24.103.126:35469/ using document root <none> and password file <none>
I20260807 11:34:29.064481 12095 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:29.064605 12095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:29.064899 12095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:29.068511 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "fa923177961c4801a81d9c6d8ddc8756"
format_stamp: "Formatted at 2026-08-07 11:34:29 on dist-test-slave-wl2h"
server_key: "ed70c33ec1e9de820a2daeb22b75fc39"
server_key_iv: "ab13d578297aa601abeceb7b960812cc"
server_key_version: "encryptionkey@0"
I20260807 11:34:29.069340 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "fa923177961c4801a81d9c6d8ddc8756"
format_stamp: "Formatted at 2026-08-07 11:34:29 on dist-test-slave-wl2h"
server_key: "ed70c33ec1e9de820a2daeb22b75fc39"
server_key_iv: "ab13d578297aa601abeceb7b960812cc"
server_key_version: "encryptionkey@0"
I20260807 11:34:29.075424 12095 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 11:34:29.079517 12111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:29.081054 12095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:34:29.081250 12095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "fa923177961c4801a81d9c6d8ddc8756"
format_stamp: "Formatted at 2026-08-07 11:34:29 on dist-test-slave-wl2h"
server_key: "ed70c33ec1e9de820a2daeb22b75fc39"
server_key_iv: "ab13d578297aa601abeceb7b960812cc"
server_key_version: "encryptionkey@0"
I20260807 11:34:29.081439 12095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:29.100703 12095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:29.105628 12095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:29.105919 12095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:29.127255 12095 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:41533
I20260807 11:34:29.127288 12166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:41533 every 8 connection(s)
I20260807 11:34:29.128764 12095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:34:29.129240 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 12095
I20260807 11:34:29.129514 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:34:29.130571 24989 external_mini_cluster.cc:1428] Setting key c75ae914ebc3f4a820078498015fd613
I20260807 11:34:29.134070 12167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:34:29 dist-test-slave-wl2h krb5kdc[12080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:29.145694 12167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756: Bootstrap starting.
I20260807 11:34:29.149369 12167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:29.150511 12167 log.cc:824] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:29.153849 12167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756: No bootstrap required, opened a new log
I20260807 11:34:29.154146 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:29.132985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.24.103.126:41533)
0807 11:34:29.133847 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:29.133859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:29.133900 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:34:29.135814 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:29.135838 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:29.136207 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:29.136613 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:29.138266 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:29.139731 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:29.141110 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:29.141799 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:29.142009 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:29.146340 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:29.146348 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:29.146361 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:29.146401 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:29.149732 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:29.150486 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:29.150493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:29.150503 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:29.150730 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:29.151208 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:29.151213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:29.151217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:29.151793 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:29.152135 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:29.152647 (+ 512us) server_negotiation.cc:299] Negotiation successful
0807 11:34:29.153084 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":379,"threads_started":1}
I20260807 11:34:29.163727 12167 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa923177961c4801a81d9c6d8ddc8756" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41533 } }
I20260807 11:34:29.164232 12167 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:29.164381 12167 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa923177961c4801a81d9c6d8ddc8756, State: Initialized, Role: FOLLOWER
I20260807 11:34:29.165187 12167 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa923177961c4801a81d9c6d8ddc8756" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41533 } }
I20260807 11:34:29.165460 12167 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:34:29.165577 12167 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:34:29.165714 12167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:29.168217 12167 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa923177961c4801a81d9c6d8ddc8756" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41533 } }
I20260807 11:34:29.168717 12167 leader_election.cc:302] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa923177961c4801a81d9c6d8ddc8756; no voters:
I20260807 11:34:29.169401 12167 leader_election.cc:288] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:34:29.169570 12172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:29.170325 12172 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [term 1 LEADER]: Becoming Leader. State: Replica: fa923177961c4801a81d9c6d8ddc8756, State: Running, Role: LEADER
I20260807 11:34:29.171080 12172 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa923177961c4801a81d9c6d8ddc8756" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41533 } }
I20260807 11:34:29.171751 12167 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:34:29.174546 12173 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa923177961c4801a81d9c6d8ddc8756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa923177961c4801a81d9c6d8ddc8756" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41533 } } }
I20260807 11:34:29.174940 12173 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:29.175799 12174 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa923177961c4801a81d9c6d8ddc8756. Latest consensus state: current_term: 1 leader_uuid: "fa923177961c4801a81d9c6d8ddc8756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa923177961c4801a81d9c6d8ddc8756" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41533 } } }
I20260807 11:34:29.176220 12174 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:29.176749 12181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:34:29.181648 12181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:34:29.195562 12181 catalog_manager.cc:1347] Generated new cluster ID: d8364e60525b41029305c5ce2533f271
I20260807 11:34:29.195706 12181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:34:29.224092 12181 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:34:29.225579 12181 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:34:29.233636 12181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756: Generated new TSK 0
I20260807 11:34:29.234586 12181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:34:29.321550 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41533
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:29.702193 12197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:29.702512 12197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:29.702662 12197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:29.710533 12197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:29.710675 12197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:29.710881 12197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:34:29.721552 12197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:29.722970 12197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:29.724480 12197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:29.739492 12203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:29.739732 12202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:29.739730 12205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:29.740691 12197 server_base.cc:1034] running on GCE node
I20260807 11:34:29.741417 12197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:29.742461 12197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:29.743716 12197 hybrid_clock.cc:648] HybridClock initialized: now 1786102469743628 us; error 58 us; skew 500 ppm
I20260807 11:34:29.744130 12197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:29 dist-test-slave-wl2h krb5kdc[12080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102469, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:29.748127 12197 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:34:29.750300 12197 webserver.cc:460] Webserver started at http://127.24.103.65:43367/ using document root <none> and password file <none>
I20260807 11:34:29.751654 12197 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:29.751909 12197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:29.752473 12197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:29.756397 12197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "68bbd864fc0846dbb4f3ea8148119132"
format_stamp: "Formatted at 2026-08-07 11:34:29 on dist-test-slave-wl2h"
server_key: "c4b699401e41fe81e51af9491b245658"
server_key_iv: "f1737449680344891a8af2620a2303ee"
server_key_version: "encryptionkey@0"
I20260807 11:34:29.757270 12197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "68bbd864fc0846dbb4f3ea8148119132"
format_stamp: "Formatted at 2026-08-07 11:34:29 on dist-test-slave-wl2h"
server_key: "c4b699401e41fe81e51af9491b245658"
server_key_iv: "f1737449680344891a8af2620a2303ee"
server_key_version: "encryptionkey@0"
I20260807 11:34:29.763406 12197 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 11:34:29.767434 12212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:29.768958 12197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:34:29.769157 12197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "68bbd864fc0846dbb4f3ea8148119132"
format_stamp: "Formatted at 2026-08-07 11:34:29 on dist-test-slave-wl2h"
server_key: "c4b699401e41fe81e51af9491b245658"
server_key_iv: "f1737449680344891a8af2620a2303ee"
server_key_version: "encryptionkey@0"
I20260807 11:34:29.769456 12197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:29.788118 12197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:29.793462 12197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:29.793853 12197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:29.795159 12197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:29.796767 12197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:29.796861 12197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:29.796998 12197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:29.797063 12197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:29.836833 12197 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:46405
I20260807 11:34:29.836931 12340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:46405 every 8 connection(s)
I20260807 11:34:29.838328 12197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:34:29.848169 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 12197
I20260807 11:34:29.848407 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:34:29.849093 24989 external_mini_cluster.cc:1428] Setting key ee9cb36a346bd4abcf30d363310e7c72
Aug 07 11:34:29 dist-test-slave-wl2h krb5kdc[12080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102469, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:29.859484 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:29.841295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:43581 (local address 127.24.103.126:41533)
0807 11:34:29.841851 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:29.841860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:29.842763 (+ 903us) server_negotiation.cc:407] Connection header received
0807 11:34:29.844747 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:29.844754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:29.844849 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:29.844973 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:29.846480 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:29.847304 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:29.848980 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:29.849444 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:29.849589 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:29.853477 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:29.853482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:29.853486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:29.853520 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:29.856231 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:29.857004 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:29.857014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:29.857017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:29.857138 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:29.857586 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:29.857591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:29.857595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:29.857881 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:29.858045 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:29.858805 (+ 760us) server_negotiation.cc:299] Negotiation successful
0807 11:34:29.859107 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":233,"threads_started":1}
I20260807 11:34:29.860680 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:29.841874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41533 (local address 127.24.103.65:43581)
0807 11:34:29.842600 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:29.842638 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:29.844458 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:29.845218 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:29.845232 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:29.845735 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:29.846305 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:29.846322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:29.847483 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:29.847487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:29.848836 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:29.848849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:29.849680 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:29.849689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:29.850002 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:29.850830 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:34:29.850865 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:29.853220 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:29.856385 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:29.856395 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:29.856413 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:29.856868 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:29.857260 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:29.857265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:29.857269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:29.857469 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:29.858016 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:29.858026 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:29.858403 (+ 377us) client_negotiation.cc:769] Sending connection context
0807 11:34:29.858686 (+ 283us) client_negotiation.cc:240] Negotiation successful
0807 11:34:29.859315 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":330,"threads_started":1}
I20260807 11:34:29.862782 12341 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41533
I20260807 11:34:29.863188 12341 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:29.864069 12341 heartbeater.cc:511] Master 127.24.103.126:41533 requested a full tablet report, sending...
I20260807 11:34:29.866358 12129 ts_manager.cc:194] Registered new tserver with Master: 68bbd864fc0846dbb4f3ea8148119132 (127.24.103.65:46405)
I20260807 11:34:29.867923 12129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:43581
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:34:29.927909 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41533
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:30.313496 12348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:30.313884 12348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:30.314090 12348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:30.322386 12348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:30.322561 12348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:30.322773 12348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:34:30.333904 12348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:30.335456 12348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:30.336910 12348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:30.352336 12354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:30.352336 12356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:30.352494 12353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:30.353400 12348 server_base.cc:1034] running on GCE node
I20260807 11:34:30.354140 12348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:30.355175 12348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:30.356427 12348 hybrid_clock.cc:648] HybridClock initialized: now 1786102470356358 us; error 65 us; skew 500 ppm
I20260807 11:34:30.356842 12348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:30 dist-test-slave-wl2h krb5kdc[12080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102470, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:30.360546 12348 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:34:30.362427 12348 webserver.cc:460] Webserver started at http://127.24.103.66:35391/ using document root <none> and password file <none>
I20260807 11:34:30.363296 12348 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:30.363466 12348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:30.363852 12348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:30.367287 12348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "9f890e4a42c74e748a3422b3b4b0fa1a"
format_stamp: "Formatted at 2026-08-07 11:34:30 on dist-test-slave-wl2h"
server_key: "a9e59cc6d4c60ff339246f83af267748"
server_key_iv: "2538fd96ed5268a2f7883c167be25424"
server_key_version: "encryptionkey@0"
I20260807 11:34:30.368193 12348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "9f890e4a42c74e748a3422b3b4b0fa1a"
format_stamp: "Formatted at 2026-08-07 11:34:30 on dist-test-slave-wl2h"
server_key: "a9e59cc6d4c60ff339246f83af267748"
server_key_iv: "2538fd96ed5268a2f7883c167be25424"
server_key_version: "encryptionkey@0"
I20260807 11:34:30.374450 12348 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 11:34:30.378858 12364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:30.380728 12348 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260807 11:34:30.380973 12348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "9f890e4a42c74e748a3422b3b4b0fa1a"
format_stamp: "Formatted at 2026-08-07 11:34:30 on dist-test-slave-wl2h"
server_key: "a9e59cc6d4c60ff339246f83af267748"
server_key_iv: "2538fd96ed5268a2f7883c167be25424"
server_key_version: "encryptionkey@0"
I20260807 11:34:30.381232 12348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:30.400040 12348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:30.405576 12348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:30.405922 12348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:30.407121 12348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:30.408735 12348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:30.408855 12348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:30.409030 12348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:30.409123 12348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:30.447544 12348 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:39587
I20260807 11:34:30.447649 12491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:39587 every 8 connection(s)
I20260807 11:34:30.449122 12348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:34:30.454231 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 12348
I20260807 11:34:30.454449 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:34:30.454957 24989 external_mini_cluster.cc:1428] Setting key 83cfb6ecfeec25d9130e45a9850c5d62
Aug 07 11:34:30 dist-test-slave-wl2h krb5kdc[12080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102470, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:30.473193 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:30.451894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:58995 (local address 127.24.103.126:41533)
0807 11:34:30.452555 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:30.452565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:30.453238 (+ 673us) server_negotiation.cc:407] Connection header received
0807 11:34:30.455452 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:30.455458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:30.455569 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:30.455704 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:30.457784 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:30.458497 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:30.461370 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:30.461835 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:30.462000 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:30.466766 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:30.466771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:30.466775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:30.466811 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:30.469719 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:30.470533 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:30.470545 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:30.470548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:30.470672 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:30.471189 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:30.471193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:30.471195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:30.471485 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:30.471654 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:30.472503 (+ 849us) server_negotiation.cc:299] Negotiation successful
0807 11:34:30.472744 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":397,"threads_started":1}
I20260807 11:34:30.474309 12495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:30.452294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41533 (local address 127.24.103.66:58995)
0807 11:34:30.453056 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:30.453107 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:30.455159 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:30.456017 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:30.456030 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:30.456680 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:30.457557 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:30.457586 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:30.458682 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:30.458692 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:30.461186 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:30.461207 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:30.462158 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:30.462169 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:30.462544 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:30.463454 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:34:30.463500 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:30.466485 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:30.469887 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:30.469897 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:30.469916 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:30.470389 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:30.470794 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:30.470799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:30.470803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:30.471002 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:30.471626 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:30.471701 (+ 75us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:30.472120 (+ 419us) client_negotiation.cc:769] Sending connection context
0807 11:34:30.472403 (+ 283us) client_negotiation.cc:240] Negotiation successful
0807 11:34:30.473071 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":322,"threads_started":1}
I20260807 11:34:30.475865 12492 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41533
I20260807 11:34:30.476276 12492 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:30.477247 12492 heartbeater.cc:511] Master 127.24.103.126:41533 requested a full tablet report, sending...
I20260807 11:34:30.479228 12129 ts_manager.cc:194] Registered new tserver with Master: 9f890e4a42c74e748a3422b3b4b0fa1a (127.24.103.66:39587)
I20260807 11:34:30.480727 12129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:58995
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:34:30.540222 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41533
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:34:30.871181 12341 heartbeater.cc:503] Master 127.24.103.126:41533 was elected leader, sending a full tablet report...
W20260807 11:34:30.913679 12499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:30.914041 12499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:30.914158 12499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:30.921633 12499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:30.921766 12499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:30.921885 12499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:34:30.932127 12499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:30.933554 12499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:30.934911 12499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:30.949854 12509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:30.949894 12506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:30.949946 12507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:30.953073 12499 server_base.cc:1034] running on GCE node
I20260807 11:34:30.953858 12499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:30.954952 12499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:30.956254 12499 hybrid_clock.cc:648] HybridClock initialized: now 1786102470956206 us; error 34 us; skew 500 ppm
I20260807 11:34:30.956702 12499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:30 dist-test-slave-wl2h krb5kdc[12080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102470, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:30.961071 12499 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:34:30.962863 12499 webserver.cc:460] Webserver started at http://127.24.103.67:46255/ using document root <none> and password file <none>
I20260807 11:34:30.963739 12499 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:30.963905 12499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:30.964295 12499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:30.967535 12499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "c46a9d86ad284a0d942fc01a1ace6d83"
format_stamp: "Formatted at 2026-08-07 11:34:30 on dist-test-slave-wl2h"
server_key: "f9d09c61ba0376113c438ef94f0e4f51"
server_key_iv: "5f025f3a8050d4432e5ae9566e32171c"
server_key_version: "encryptionkey@0"
I20260807 11:34:30.968392 12499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "c46a9d86ad284a0d942fc01a1ace6d83"
format_stamp: "Formatted at 2026-08-07 11:34:30 on dist-test-slave-wl2h"
server_key: "f9d09c61ba0376113c438ef94f0e4f51"
server_key_iv: "5f025f3a8050d4432e5ae9566e32171c"
server_key_version: "encryptionkey@0"
I20260807 11:34:30.974786 12499 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 11:34:30.978926 12517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:30.980659 12499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:34:30.980866 12499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "c46a9d86ad284a0d942fc01a1ace6d83"
format_stamp: "Formatted at 2026-08-07 11:34:30 on dist-test-slave-wl2h"
server_key: "f9d09c61ba0376113c438ef94f0e4f51"
server_key_iv: "5f025f3a8050d4432e5ae9566e32171c"
server_key_version: "encryptionkey@0"
I20260807 11:34:30.981114 12499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:31.015511 12499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:31.021096 12499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:31.021477 12499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:31.022567 12499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:31.024055 12499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:31.024169 12499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:31.024338 12499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:31.024423 12499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:31.061240 12499 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:44073
I20260807 11:34:31.061298 12649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:44073 every 8 connection(s)
I20260807 11:34:31.062597 12499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:34:31.068861 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 12499
I20260807 11:34:31.069093 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:34:31.069589 24989 external_mini_cluster.cc:1428] Setting key d3fab64b90295c3b1669a4d36524657b
Aug 07 11:34:31 dist-test-slave-wl2h krb5kdc[12080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102470, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:31.086053 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.065408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:41265 (local address 127.24.103.126:41533)
0807 11:34:31.065970 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:31.065975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:31.066741 (+ 766us) server_negotiation.cc:407] Connection header received
0807 11:34:31.068726 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:31.068732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:31.068828 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:31.068949 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:31.070775 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.071995 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.074605 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.075151 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.075296 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.079071 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:31.079079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:31.079087 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:31.079123 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:31.082176 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:31.082994 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:31.082998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:31.083001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:31.083240 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:31.083680 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:31.083685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:31.083687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:31.084025 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:31.084262 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:31.085383 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0807 11:34:31.085634 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":305,"threads_started":1}
I20260807 11:34:31.087494 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.065787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41533 (local address 127.24.103.67:41265)
0807 11:34:31.066578 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:31.066616 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:31.068414 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:31.069255 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:31.069264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:31.069833 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:31.070546 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.070567 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.072282 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.072291 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.074382 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.074396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.075408 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.075413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.075740 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.076504 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:34:31.076527 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:31.078782 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:31.082338 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:31.082346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:31.082360 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:31.082854 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:31.083360 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:31.083363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:31.083365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:31.083555 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:31.084250 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:31.084262 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:31.084884 (+ 622us) client_negotiation.cc:769] Sending connection context
0807 11:34:31.085236 (+ 352us) client_negotiation.cc:240] Negotiation successful
0807 11:34:31.085838 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":337,"threads_started":1}
I20260807 11:34:31.089407 12652 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41533
I20260807 11:34:31.089743 12652 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:31.090508 12652 heartbeater.cc:511] Master 127.24.103.126:41533 requested a full tablet report, sending...
I20260807 11:34:31.092267 12129 ts_manager.cc:194] Registered new tserver with Master: c46a9d86ad284a0d942fc01a1ace6d83 (127.24.103.67:44073)
I20260807 11:34:31.093154 12129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:41265
I20260807 11:34:31.100000 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:34:31.119174 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.105232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.24.103.126:41533)
0807 11:34:31.105443 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:31.105448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:31.106038 (+ 590us) server_negotiation.cc:407] Connection header received
0807 11:34:31.106198 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:31.106203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:31.106275 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:31.106371 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:31.107969 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.108782 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.110125 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.110579 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.110718 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.113484 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:31.113489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:31.113493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:31.113529 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:31.116517 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:31.117255 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:31.117259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:31.117261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:31.117395 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:31.117753 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:31.117757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:31.117759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:31.118107 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:31.118342 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:31.118665 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 11:34:31.118858 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 11:34:31.123807 12129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:34:31.131978 12129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:34:31.166824 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.148090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.24.103.65:46405)
0807 11:34:31.155833 (+ 7743us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:31.155847 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:31.157067 (+ 1220us) server_negotiation.cc:407] Connection header received
0807 11:34:31.157540 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:31.157552 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:31.157830 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:31.158050 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:31.159281 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.160466 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.163598 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.165165 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.165339 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.165613 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:31.166043 (+ 430us) server_negotiation.cc:299] Negotiation successful
0807 11:34:31.166253 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7497,"thread_start_us":7282,"threads_started":1}
I20260807 11:34:31.167390 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.146075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:44073 (local address 127.0.0.1:58584)
0807 11:34:31.150740 (+ 4665us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:31.150791 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:31.151002 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:31.158617 (+ 7615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:31.158622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:31.158657 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:31.159186 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.159199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.161634 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.161643 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.164097 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.164111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.166161 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.166169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.166360 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.166404 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 11:34:31.166491 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 11:34:31.166640 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2645,"mutex_wait_us":103,"thread_start_us":9945,"threads_started":1}
I20260807 11:34:31.167606 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.149575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:39587 (local address 127.0.0.1:45096)
0807 11:34:31.157076 (+ 7501us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:31.157098 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:31.157244 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:31.158086 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:31.158094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:31.158136 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:31.158617 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.158633 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.161125 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.161131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.163596 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.163613 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.165275 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.165281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.165486 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.165550 (+ 64us) client_negotiation.cc:769] Sending connection context
0807 11:34:31.165694 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 11:34:31.167173 (+ 1479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7387,"mutex_wait_us":6,"spinlock_wait_cycles":17152,"thread_start_us":7218,"threads_started":1}
I20260807 11:34:31.167685 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.146348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.24.103.67:44073)
0807 11:34:31.158062 (+ 11714us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:31.158070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:31.158102 (+ 32us) server_negotiation.cc:407] Connection header received
0807 11:34:31.158193 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:31.158199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:31.158402 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:31.158600 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:31.159427 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.161388 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.164292 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.165850 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.166766 (+ 916us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.167058 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:31.167168 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 11:34:31.167321 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11520,"thread_start_us":11347,"threads_started":1}
I20260807 11:34:31.166646 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.146761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:46405 (local address 127.0.0.1:47186)
0807 11:34:31.156317 (+ 9556us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:31.156337 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:31.157274 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:31.158365 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:31.158371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:31.158394 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:31.158991 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.159002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.160610 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.160618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.163381 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.163398 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.165636 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.165644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.165811 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.165842 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 11:34:31.165926 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 11:34:31.166141 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9419,"mutex_wait_us":107,"spinlock_wait_cycles":896,"thread_start_us":1767,"threads_started":1}
I20260807 11:34:31.168215 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.150034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.24.103.66:39587)
0807 11:34:31.156625 (+ 6591us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:31.156633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:31.157376 (+ 743us) server_negotiation.cc:407] Connection header received
0807 11:34:31.157490 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:31.157499 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:31.157835 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:31.158049 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:31.158875 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.160902 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.163774 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.165072 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.165237 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.165452 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:31.166970 (+ 1518us) server_negotiation.cc:299] Negotiation successful
0807 11:34:31.167940 (+ 970us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6429,"thread_start_us":6322,"threads_started":1}
I20260807 11:34:31.171738 12267 tablet_service.cc:1467] Processing CreateTablet for tablet 3023109e10e34203b32f5c5fb1789fa8 (DEFAULT_TABLE table=test-table [id=699c90f9be25420a80dea0269977437f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:31.171707 12576 tablet_service.cc:1467] Processing CreateTablet for tablet 3023109e10e34203b32f5c5fb1789fa8 (DEFAULT_TABLE table=test-table [id=699c90f9be25420a80dea0269977437f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:31.171718 12418 tablet_service.cc:1467] Processing CreateTablet for tablet 3023109e10e34203b32f5c5fb1789fa8 (DEFAULT_TABLE table=test-table [id=699c90f9be25420a80dea0269977437f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:31.173264 12267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3023109e10e34203b32f5c5fb1789fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:31.173260 12418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3023109e10e34203b32f5c5fb1789fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:31.173267 12576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3023109e10e34203b32f5c5fb1789fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:31.187589 12678 tablet_bootstrap.cc:492] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83: Bootstrap starting.
I20260807 11:34:31.189651 12679 tablet_bootstrap.cc:492] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a: Bootstrap starting.
I20260807 11:34:31.192080 12678 tablet_bootstrap.cc:654] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:31.192826 12680 tablet_bootstrap.cc:492] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132: Bootstrap starting.
I20260807 11:34:31.193452 12678 log.cc:824] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:31.193534 12679 tablet_bootstrap.cc:654] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:31.194947 12679 log.cc:824] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:31.196166 12680 tablet_bootstrap.cc:654] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:31.196965 12678 tablet_bootstrap.cc:492] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83: No bootstrap required, opened a new log
I20260807 11:34:31.197263 12678 ts_tablet_manager.cc:1397] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 11:34:31.197379 12680 log.cc:824] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:31.198030 12679 tablet_bootstrap.cc:492] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a: No bootstrap required, opened a new log
I20260807 11:34:31.198343 12679 ts_tablet_manager.cc:1397] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 11:34:31.200677 12680 tablet_bootstrap.cc:492] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132: No bootstrap required, opened a new log
I20260807 11:34:31.200930 12680 ts_tablet_manager.cc:1397] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 11:34:31.207319 12679 raft_consensus.cc:348] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44073 } } peers { permanent_uuid: "68bbd864fc0846dbb4f3ea8148119132" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46405 } } peers { permanent_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39587 } }
I20260807 11:34:31.207789 12679 raft_consensus.cc:374] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:31.207903 12679 raft_consensus.cc:729] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f890e4a42c74e748a3422b3b4b0fa1a, State: Initialized, Role: FOLLOWER
I20260807 11:34:31.208032 12678 raft_consensus.cc:348] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44073 } } peers { permanent_uuid: "68bbd864fc0846dbb4f3ea8148119132" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46405 } } peers { permanent_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39587 } }
I20260807 11:34:31.208469 12678 raft_consensus.cc:374] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:31.208582 12678 raft_consensus.cc:729] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46a9d86ad284a0d942fc01a1ace6d83, State: Initialized, Role: FOLLOWER
I20260807 11:34:31.208693 12679 consensus_queue.cc:260] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44073 } } peers { permanent_uuid: "68bbd864fc0846dbb4f3ea8148119132" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46405 } } peers { permanent_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39587 } }
I20260807 11:34:31.209156 12678 consensus_queue.cc:260] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44073 } } peers { permanent_uuid: "68bbd864fc0846dbb4f3ea8148119132" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46405 } } peers { permanent_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39587 } }
I20260807 11:34:31.210196 12492 heartbeater.cc:503] Master 127.24.103.126:41533 was elected leader, sending a full tablet report...
I20260807 11:34:31.210975 12679 ts_tablet_manager.cc:1428] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260807 11:34:31.212133 12652 heartbeater.cc:503] Master 127.24.103.126:41533 was elected leader, sending a full tablet report...
I20260807 11:34:31.212827 12678 ts_tablet_manager.cc:1428] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260807 11:34:31.212510 12680 raft_consensus.cc:348] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44073 } } peers { permanent_uuid: "68bbd864fc0846dbb4f3ea8148119132" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46405 } } peers { permanent_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39587 } }
I20260807 11:34:31.212980 12680 raft_consensus.cc:374] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:31.213099 12680 raft_consensus.cc:729] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bbd864fc0846dbb4f3ea8148119132, State: Initialized, Role: FOLLOWER
I20260807 11:34:31.213567 12686 raft_consensus.cc:482] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:34:31.213738 12680 consensus_queue.cc:260] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44073 } } peers { permanent_uuid: "68bbd864fc0846dbb4f3ea8148119132" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46405 } } peers { permanent_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39587 } }
I20260807 11:34:31.213882 12686 raft_consensus.cc:504] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44073 } } peers { permanent_uuid: "68bbd864fc0846dbb4f3ea8148119132" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46405 } } peers { permanent_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39587 } }
I20260807 11:34:31.220687 12686 leader_election.cc:288] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68bbd864fc0846dbb4f3ea8148119132 (127.24.103.65:46405), 9f890e4a42c74e748a3422b3b4b0fa1a (127.24.103.66:39587)
I20260807 11:34:31.222641 12680 ts_tablet_manager.cc:1428] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132: Time spent starting tablet: real 0.022s user 0.006s sys 0.016s
I20260807 11:34:31.229378 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.222267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:60513 (local address 127.24.103.65:46405)
0807 11:34:31.223128 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:31.223134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:31.223156 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:34:31.223221 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:31.223225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:31.223293 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:31.223392 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:31.223938 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.225174 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.227593 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.228413 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.228545 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.228596 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:31.229085 (+ 489us) server_negotiation.cc:299] Negotiation successful
0807 11:34:31.229189 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683}
I20260807 11:34:31.229519 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.222625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:46405 (local address 127.24.103.67:60513)
0807 11:34:31.222829 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:31.222851 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:31.222978 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:31.223575 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:31.223581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:31.223607 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:31.223787 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.223795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.225340 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.225348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.227355 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.227371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.228588 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.228595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.228878 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.228926 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 11:34:31.229047 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 11:34:31.229198 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":24}
I20260807 11:34:31.230484 12290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3023109e10e34203b32f5c5fb1789fa8" candidate_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68bbd864fc0846dbb4f3ea8148119132" is_pre_election: true
I20260807 11:34:31.231055 12290 raft_consensus.cc:2454] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c46a9d86ad284a0d942fc01a1ace6d83 in term 0.
I20260807 11:34:31.231746 12521 leader_election.cc:302] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68bbd864fc0846dbb4f3ea8148119132, c46a9d86ad284a0d942fc01a1ace6d83; no voters:
I20260807 11:34:31.232219 12687 raft_consensus.cc:2781] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:34:31.232473 12687 raft_consensus.cc:482] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:34:31.232609 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.224195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:39587 (local address 127.24.103.67:36019)
0807 11:34:31.225119 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:31.225150 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:31.225346 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:31.225957 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:31.225962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:31.225992 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:31.226549 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.226575 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.227975 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.227980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.230068 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:31.230089 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.231508 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.231514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:31.231714 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.231744 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 11:34:31.231867 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 11:34:31.232037 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":27,"thread_start_us":601,"threads_started":1}
I20260807 11:34:31.232703 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.224348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:36019 (local address 127.24.103.66:39587)
0807 11:34:31.224598 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:31.224604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:31.225229 (+ 625us) server_negotiation.cc:407] Connection header received
0807 11:34:31.225537 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:31.225543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:31.225639 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:31.225709 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:31.226742 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.227818 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.230247 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.231309 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.231502 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.231590 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:31.231891 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 11:34:31.232403 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 11:34:31.232970 12687 raft_consensus.cc:3037] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:31.233750 12441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3023109e10e34203b32f5c5fb1789fa8" candidate_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" is_pre_election: true
I20260807 11:34:31.234265 12441 raft_consensus.cc:2454] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c46a9d86ad284a0d942fc01a1ace6d83 in term 0.
I20260807 11:34:31.235641 12687 raft_consensus.cc:504] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44073 } } peers { permanent_uuid: "68bbd864fc0846dbb4f3ea8148119132" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46405 } } peers { permanent_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39587 } }
I20260807 11:34:31.236239 12687 leader_election.cc:288] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [CANDIDATE]: Term 1 election: Requested vote from peers 68bbd864fc0846dbb4f3ea8148119132 (127.24.103.65:46405), 9f890e4a42c74e748a3422b3b4b0fa1a (127.24.103.66:39587)
I20260807 11:34:31.236754 12290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3023109e10e34203b32f5c5fb1789fa8" candidate_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68bbd864fc0846dbb4f3ea8148119132"
I20260807 11:34:31.236974 12290 raft_consensus.cc:3037] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:31.237182 12441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3023109e10e34203b32f5c5fb1789fa8" candidate_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a"
I20260807 11:34:31.237404 12441 raft_consensus.cc:3037] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:31.239373 12290 raft_consensus.cc:2454] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c46a9d86ad284a0d942fc01a1ace6d83 in term 1.
I20260807 11:34:31.239809 12521 leader_election.cc:302] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68bbd864fc0846dbb4f3ea8148119132, c46a9d86ad284a0d942fc01a1ace6d83; no voters:
I20260807 11:34:31.240114 12687 raft_consensus.cc:2781] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:31.240089 12441 raft_consensus.cc:2454] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c46a9d86ad284a0d942fc01a1ace6d83 in term 1.
I20260807 11:34:31.240295 12687 raft_consensus.cc:686] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 1 LEADER]: Becoming Leader. State: Replica: c46a9d86ad284a0d942fc01a1ace6d83, State: Running, Role: LEADER
I20260807 11:34:31.240778 12687 consensus_queue.cc:237] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44073 } } peers { permanent_uuid: "68bbd864fc0846dbb4f3ea8148119132" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46405 } } peers { permanent_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39587 } }
I20260807 11:34:31.245370 12129 catalog_manager.cc:5495] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 reported cstate change: term changed from 0 to 1, leader changed from <none> to c46a9d86ad284a0d942fc01a1ace6d83 (127.24.103.67). New cstate: current_term: 1 leader_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46a9d86ad284a0d942fc01a1ace6d83" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 44073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68bbd864fc0846dbb4f3ea8148119132" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39587 } health_report { overall_health: UNKNOWN } } }
I20260807 11:34:31.265061 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.259604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.24.103.67:44073)
0807 11:34:31.259796 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:31.259800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:31.259913 (+ 113us) server_negotiation.cc:407] Connection header received
0807 11:34:31.260087 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:31.260091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:31.260191 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:31.260321 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:34:31.260719 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.261459 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.262875 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.263331 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.263422 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.263917 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:34:31.264455 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:34:31.264602 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:31.264784 (+ 182us) server_negotiation.cc:299] Negotiation successful
0807 11:34:31.264851 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 11:34:31.273268 12441 raft_consensus.cc:1261] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a [term 1 FOLLOWER]: Refusing update from remote peer c46a9d86ad284a0d942fc01a1ace6d83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:31.273327 12290 raft_consensus.cc:1261] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132 [term 1 FOLLOWER]: Refusing update from remote peer c46a9d86ad284a0d942fc01a1ace6d83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:31.274331 12686 consensus_queue.cc:1035] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f890e4a42c74e748a3422b3b4b0fa1a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:34:31.274710 12687 consensus_queue.cc:1035] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68bbd864fc0846dbb4f3ea8148119132" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 46405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:34:31.286626 12695 mvcc.cc:205] Tried to move back new op lower bound from 7315875722324631552 to 7315875722206298112. Current Snapshot: MvccSnapshot[applied={T|T < 7315875722324631552}]
I20260807 11:34:31.289203 12696 mvcc.cc:205] Tried to move back new op lower bound from 7315875722324631552 to 7315875722206298112. Current Snapshot: MvccSnapshot[applied={T|T < 7315875722324631552}]
I20260807 11:34:31.302350 12128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:31.302693 12128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:31.307315 12128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa923177961c4801a81d9c6d8ddc8756: Sending DeleteTablet for 3 replicas of tablet 3023109e10e34203b32f5c5fb1789fa8
I20260807 11:34:31.308529 12576 tablet_service.cc:1514] Processing DeleteTablet for tablet 3023109e10e34203b32f5c5fb1789fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:31 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:58584
I20260807 11:34:31.308874 12267 tablet_service.cc:1514] Processing DeleteTablet for tablet 3023109e10e34203b32f5c5fb1789fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:31 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:47186
I20260807 11:34:31.308926 12418 tablet_service.cc:1514] Processing DeleteTablet for tablet 3023109e10e34203b32f5c5fb1789fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:31 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:45096
I20260807 11:34:31.310954 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 12197
I20260807 11:34:31.332727 12706 tablet_replica.cc:330] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83: stopping tablet replica
I20260807 11:34:31.333583 12707 tablet_replica.cc:330] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132: stopping tablet replica
I20260807 11:34:31.334246 12707 raft_consensus.cc:2229] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:31.335053 12707 raft_consensus.cc:2258] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:31.336005 12706 raft_consensus.cc:2229] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:34:31.337085 12706 raft_consensus.cc:2258] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:31.341974 12707 ts_tablet_manager.cc:1905] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:31.343575 12709 tablet_replica.cc:330] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a: stopping tablet replica
I20260807 11:34:31.344581 12706 ts_tablet_manager.cc:1905] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 11:34:31 dist-test-slave-wl2h krb5kdc[12080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:34:31.345165 12709 raft_consensus.cc:2229] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:31.345695 12709 raft_consensus.cc:2258] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:31.350155 12709 ts_tablet_manager.cc:1905] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:31.351681 12707 ts_tablet_manager.cc:1918] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:31.351907 12707 log.cc:1196] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/3023109e10e34203b32f5c5fb1789fa8
I20260807 11:34:31.352567 12707 ts_tablet_manager.cc:1939] T 3023109e10e34203b32f5c5fb1789fa8 P 68bbd864fc0846dbb4f3ea8148119132: Deleting consensus metadata
I20260807 11:34:31.354595 12115 catalog_manager.cc:4841] TS 68bbd864fc0846dbb4f3ea8148119132 (127.24.103.65:46405): tablet 3023109e10e34203b32f5c5fb1789fa8 (table test-table [id=699c90f9be25420a80dea0269977437f]) successfully deleted
I20260807 11:34:31.355172 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.312992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.24.103.65:46405)
0807 11:34:31.313251 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:31.313260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:31.323656 (+ 10396us) server_negotiation.cc:407] Connection header received
0807 11:34:31.324282 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:31.324293 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:31.324421 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:31.324540 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:31.327164 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.328982 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.337657 (+ 8675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.338711 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.338862 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.346406 (+ 7544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:31.346422 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:31.346442 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:31.346498 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:31.350841 (+ 4343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:31.351899 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:31.351909 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:31.351913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:31.352252 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:31.352858 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:31.352866 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:31.352872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:31.353394 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:31.353714 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:31.354472 (+ 758us) server_negotiation.cc:299] Negotiation successful
0807 11:34:31.354740 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 11:34:31.356325 12334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:31.358398 12706 ts_tablet_manager.cc:1918] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:31.358575 12706 log.cc:1196] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/3023109e10e34203b32f5c5fb1789fa8
I20260807 11:34:31.359287 12706 ts_tablet_manager.cc:1939] T 3023109e10e34203b32f5c5fb1789fa8 P c46a9d86ad284a0d942fc01a1ace6d83: Deleting consensus metadata
I20260807 11:34:31.359418 12709 ts_tablet_manager.cc:1918] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:31.359629 12709 log.cc:1196] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/3023109e10e34203b32f5c5fb1789fa8
I20260807 11:34:31.360208 12709 ts_tablet_manager.cc:1939] T 3023109e10e34203b32f5c5fb1789fa8 P 9f890e4a42c74e748a3422b3b4b0fa1a: Deleting consensus metadata
I20260807 11:34:31.361219 12113 catalog_manager.cc:4841] TS c46a9d86ad284a0d942fc01a1ace6d83 (127.24.103.67:44073): tablet 3023109e10e34203b32f5c5fb1789fa8 (table test-table [id=699c90f9be25420a80dea0269977437f]) successfully deleted
I20260807 11:34:31.361944 12115 catalog_manager.cc:4841] TS 9f890e4a42c74e748a3422b3b4b0fa1a (127.24.103.66:39587): tablet 3023109e10e34203b32f5c5fb1789fa8 (table test-table [id=699c90f9be25420a80dea0269977437f]) successfully deleted
I20260807 11:34:31.461719 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 12197
I20260807 11:34:31.488955 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 12348
Aug 07 11:34:31 dist-test-slave-wl2h krb5kdc[12080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:34:31.508785 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.489852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.24.103.66:39587)
0807 11:34:31.490046 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:31.490052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:31.490075 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:34:31.490234 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:31.490240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:31.490324 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:31.490441 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:31.492172 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.493357 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.494859 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.495662 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.495804 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.501363 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:31.501384 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:31.501405 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:31.501464 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:31.505540 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:31.506390 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:31.506400 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:31.506404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:31.506575 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:31.506958 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:31.506968 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:31.506972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:31.507531 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:31.507858 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:31.508174 (+ 316us) server_negotiation.cc:299] Negotiation successful
0807 11:34:31.508409 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:34:31.509673 12486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:31.607223 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 12348
I20260807 11:34:31.628134 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 12499
Aug 07 11:34:31 dist-test-slave-wl2h krb5kdc[12080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:34:31.650301 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:31.629109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.24.103.67:44073)
0807 11:34:31.629282 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:31.629287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:31.629305 (+ 18us) server_negotiation.cc:407] Connection header received
0807 11:34:31.629451 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:31.629455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:31.629520 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:31.629600 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:31.631566 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.632461 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.634598 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:31.635374 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:31.635499 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:31.641389 (+ 5890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:31.641410 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:31.641433 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:31.641484 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:31.646040 (+ 4556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:31.647790 (+ 1750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:31.647800 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:31.647803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:31.648028 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:31.648600 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:31.648605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:31.648609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:31.649138 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:31.649373 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:31.649782 (+ 409us) server_negotiation.cc:299] Negotiation successful
0807 11:34:31.649955 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 11:34:31.651201 12645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:31.777292 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 12499
I20260807 11:34:31.801179 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 12095
I20260807 11:34:31.802492 12161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:31.916888 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 12095
2026-08-07T11:34:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4872 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T11:34:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:34:31Z Disabled control of system clock
I20260807 11:34:32.022792 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44055
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:44055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:34:32.374037 12718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:32.374331 12718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:32.374414 12718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:32.381750 12718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:34:32.381872 12718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:32.381912 12718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:32.381963 12718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:34:32.382010 12718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:34:32.392058 12718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:44055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:44055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:32.393426 12718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:32.395473 12718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:32.408277 12723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:32.408382 12726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:32.408408 12724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:32.409502 12718 server_base.cc:1034] running on GCE node
I20260807 11:34:32.410279 12718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:32.411748 12718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:32.412954 12718 hybrid_clock.cc:648] HybridClock initialized: now 1786102472412894 us; error 55 us; skew 500 ppm
I20260807 11:34:32.413326 12718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:32.415656 12718 webserver.cc:460] Webserver started at http://127.24.103.126:43915/ using document root <none> and password file <none>
I20260807 11:34:32.416353 12718 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:32.416467 12718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:32.416810 12718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:32.419853 12718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "e6a9caa28efd4e66a18ebe844c4defd3"
format_stamp: "Formatted at 2026-08-07 11:34:32 on dist-test-slave-wl2h"
server_key: "6b6c17046fb6d6b2885d0f3c1a7e6b05"
server_key_iv: "3824e00b04dd96e9796fed200c04746d"
server_key_version: "encryptionkey@0"
I20260807 11:34:32.420686 12718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "e6a9caa28efd4e66a18ebe844c4defd3"
format_stamp: "Formatted at 2026-08-07 11:34:32 on dist-test-slave-wl2h"
server_key: "6b6c17046fb6d6b2885d0f3c1a7e6b05"
server_key_iv: "3824e00b04dd96e9796fed200c04746d"
server_key_version: "encryptionkey@0"
I20260807 11:34:32.426003 12718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:34:32.429633 12733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:32.431214 12718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:34:32.431409 12718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "e6a9caa28efd4e66a18ebe844c4defd3"
format_stamp: "Formatted at 2026-08-07 11:34:32 on dist-test-slave-wl2h"
server_key: "6b6c17046fb6d6b2885d0f3c1a7e6b05"
server_key_iv: "3824e00b04dd96e9796fed200c04746d"
server_key_version: "encryptionkey@0"
I20260807 11:34:32.431645 12718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:32.458613 12718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:32.459682 12718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:32.460007 12718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:32.480593 12718 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:44055
I20260807 11:34:32.480631 12790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:44055 every 8 connection(s)
I20260807 11:34:32.482082 12718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:34:32.485741 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 12718
I20260807 11:34:32.485965 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:34:32.486351 12791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:32.486558 24989 external_mini_cluster.cc:1428] Setting key 41463d2e459cfc98a27725163054412f
I20260807 11:34:32.501390 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:32.488609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.24.103.126:44055)
0807 11:34:32.489712 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:32.489724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:32.489775 (+ 51us) server_negotiation.cc:407] Connection header received
0807 11:34:32.491719 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:32.491743 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:32.492259 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:32.492602 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:32.493306 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:32.494719 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:32.496662 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:32.497355 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:32.497510 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:32.498893 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:32.498901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:32.498919 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:32.499608 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:32.499712 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:32.499734 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:32.499826 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:32.500073 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 11:34:32.500426 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":417,"threads_started":1}
I20260807 11:34:32.501605 12791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3: Bootstrap starting.
I20260807 11:34:32.505240 12791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:32.506294 12791 log.cc:824] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:32.509212 12791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3: No bootstrap required, opened a new log
I20260807 11:34:32.517505 12791 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a9caa28efd4e66a18ebe844c4defd3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44055 } }
I20260807 11:34:32.517935 12791 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:32.518051 12791 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a9caa28efd4e66a18ebe844c4defd3, State: Initialized, Role: FOLLOWER
I20260807 11:34:32.518616 12791 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a9caa28efd4e66a18ebe844c4defd3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44055 } }
I20260807 11:34:32.518883 12791 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:34:32.519069 12791 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:34:32.519280 12791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:32.521836 12791 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a9caa28efd4e66a18ebe844c4defd3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44055 } }
I20260807 11:34:32.522363 12791 leader_election.cc:302] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6a9caa28efd4e66a18ebe844c4defd3; no voters:
I20260807 11:34:32.522976 12791 leader_election.cc:288] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:34:32.523196 12796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:32.523985 12796 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [term 1 LEADER]: Becoming Leader. State: Replica: e6a9caa28efd4e66a18ebe844c4defd3, State: Running, Role: LEADER
I20260807 11:34:32.524621 12796 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a9caa28efd4e66a18ebe844c4defd3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44055 } }
I20260807 11:34:32.525218 12791 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:34:32.527905 12798 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6a9caa28efd4e66a18ebe844c4defd3. Latest consensus state: current_term: 1 leader_uuid: "e6a9caa28efd4e66a18ebe844c4defd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a9caa28efd4e66a18ebe844c4defd3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44055 } } }
I20260807 11:34:32.528559 12798 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:32.527935 12797 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6a9caa28efd4e66a18ebe844c4defd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a9caa28efd4e66a18ebe844c4defd3" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 44055 } } }
I20260807 11:34:32.529450 12797 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:32.535635 12806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:34:32.540959 12806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:34:32.549229 12806 catalog_manager.cc:1347] Generated new cluster ID: b2e713861e3d437cb5086a1ae24e3212
I20260807 11:34:32.549368 12806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:34:32.568848 12806 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:34:32.570854 12806 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:34:32.590118 12806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3: Generated new TSK 0
I20260807 11:34:32.591307 12806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:34:32.600649 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44055
--builtin_ntp_servers=127.24.103.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:34:32.972438 12816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:32.972756 12816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:32.972846 12816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:32.980697 12816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:32.980829 12816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:32.980963 12816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:34:32.990693 12816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:32.992175 12816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:32.994019 12816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:33.009061 12823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:33.009104 12822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:33.009135 12826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:33.013098 12816 server_base.cc:1034] running on GCE node
I20260807 11:34:33.013887 12816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:33.015067 12816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:33.016371 12816 hybrid_clock.cc:648] HybridClock initialized: now 1786102473016298 us; error 72 us; skew 500 ppm
I20260807 11:34:33.016821 12816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:33.019696 12816 webserver.cc:460] Webserver started at http://127.24.103.65:39141/ using document root <none> and password file <none>
I20260807 11:34:33.020489 12816 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:33.020645 12816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:33.021015 12816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:33.024386 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "d1373c99fc82495fbb3125bcfdeff5f0"
format_stamp: "Formatted at 2026-08-07 11:34:33 on dist-test-slave-wl2h"
server_key: "06823691806a6868a45445ddb27ab0a8"
server_key_iv: "f13af82d2e01f6e8a030e74f1c268d81"
server_key_version: "encryptionkey@0"
I20260807 11:34:33.025350 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "d1373c99fc82495fbb3125bcfdeff5f0"
format_stamp: "Formatted at 2026-08-07 11:34:33 on dist-test-slave-wl2h"
server_key: "06823691806a6868a45445ddb27ab0a8"
server_key_iv: "f13af82d2e01f6e8a030e74f1c268d81"
server_key_version: "encryptionkey@0"
I20260807 11:34:33.031659 12816 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 11:34:33.037467 12835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:33.039314 12816 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:34:33.039522 12816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "d1373c99fc82495fbb3125bcfdeff5f0"
format_stamp: "Formatted at 2026-08-07 11:34:33 on dist-test-slave-wl2h"
server_key: "06823691806a6868a45445ddb27ab0a8"
server_key_iv: "f13af82d2e01f6e8a030e74f1c268d81"
server_key_version: "encryptionkey@0"
I20260807 11:34:33.039803 12816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:33.067117 12816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:33.068045 12816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:33.068347 12816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:33.069491 12816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:33.071231 12816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:33.071331 12816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:33.071413 12816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:33.071477 12816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:33.109354 12816 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:34095
I20260807 11:34:33.109470 12960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:34095 every 8 connection(s)
I20260807 11:34:33.110776 12816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:34:33.117898 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 12816
I20260807 11:34:33.118119 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:34:33.118865 24989 external_mini_cluster.cc:1428] Setting key 2ca81cbbaa4042428e7e6ff798509a82
I20260807 11:34:33.122498 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44055
--builtin_ntp_servers=127.24.103.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:34:33.126406 12963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:33.113690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:42681 (local address 127.24.103.126:44055)
0807 11:34:33.114228 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:33.114233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:33.115503 (+ 1270us) server_negotiation.cc:407] Connection header received
0807 11:34:33.117355 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:33.117360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:33.117436 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:33.117541 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:33.119354 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:33.120277 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:33.122431 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:33.123293 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:33.123468 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:33.125119 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:33.125128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:33.125137 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:33.125167 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:33.125227 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:33.125248 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:33.125353 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:33.125652 (+ 299us) server_negotiation.cc:299] Negotiation successful
0807 11:34:33.125810 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":245,"threads_started":1}
I20260807 11:34:33.127701 12966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:33.114320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44055 (local address 127.24.103.65:42681)
0807 11:34:33.115331 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:33.115372 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:33.117055 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:33.117841 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:33.117854 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:33.117983 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:33.119073 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:33.119102 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:33.120489 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:33.120495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:33.122147 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:33.122161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:33.123555 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:33.123561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:33.123822 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:33.124688 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:34:33.124691 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:33.124747 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:34:33.124846 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:33.125373 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:33.125409 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:33.125421 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 11:34:33.125599 (+ 178us) client_negotiation.cc:240] Negotiation successful
0807 11:34:33.125992 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":278,"threads_started":1}
I20260807 11:34:33.129475 12961 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44055
I20260807 11:34:33.129828 12961 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:33.130663 12961 heartbeater.cc:511] Master 127.24.103.126:44055 requested a full tablet report, sending...
I20260807 11:34:33.133033 12752 ts_manager.cc:194] Registered new tserver with Master: d1373c99fc82495fbb3125bcfdeff5f0 (127.24.103.65:34095)
I20260807 11:34:33.134639 12752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.65:42681
W20260807 11:34:33.492466 12967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:33.492915 12967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:33.493044 12967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:33.503305 12967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:33.503443 12967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:33.503624 12967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:34:33.513491 12967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:33.515161 12967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:33.517083 12967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:33.530965 12972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:33.531260 12973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:33.531216 12976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:33.532958 12967 server_base.cc:1034] running on GCE node
I20260807 11:34:33.533679 12967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:33.534814 12967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:33.536053 12967 hybrid_clock.cc:648] HybridClock initialized: now 1786102473535967 us; error 72 us; skew 500 ppm
I20260807 11:34:33.536489 12967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:33.539382 12967 webserver.cc:460] Webserver started at http://127.24.103.66:40029/ using document root <none> and password file <none>
I20260807 11:34:33.540246 12967 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:33.540401 12967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:33.540759 12967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:33.544152 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "45132ac8b6c9470fbdc79840c92cb277"
format_stamp: "Formatted at 2026-08-07 11:34:33 on dist-test-slave-wl2h"
server_key: "8c345684c50675981ee90572c87fa6e5"
server_key_iv: "ac419790dbae4149b3545c16bdbdffa6"
server_key_version: "encryptionkey@0"
I20260807 11:34:33.545111 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "45132ac8b6c9470fbdc79840c92cb277"
format_stamp: "Formatted at 2026-08-07 11:34:33 on dist-test-slave-wl2h"
server_key: "8c345684c50675981ee90572c87fa6e5"
server_key_iv: "ac419790dbae4149b3545c16bdbdffa6"
server_key_version: "encryptionkey@0"
I20260807 11:34:33.550837 12967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 11:34:33.554849 12983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:33.557278 12967 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 11:34:33.557535 12967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "45132ac8b6c9470fbdc79840c92cb277"
format_stamp: "Formatted at 2026-08-07 11:34:33 on dist-test-slave-wl2h"
server_key: "8c345684c50675981ee90572c87fa6e5"
server_key_iv: "ac419790dbae4149b3545c16bdbdffa6"
server_key_version: "encryptionkey@0"
I20260807 11:34:33.557826 12967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:33.586024 12967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:33.587044 12967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:33.587394 12967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:33.588500 12967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:33.590216 12967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:33.590353 12967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:33.590474 12967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:33.590585 12967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:33.627488 12967 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:35785
I20260807 11:34:33.627576 13108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:35785 every 8 connection(s)
I20260807 11:34:33.628914 12967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:34:33.630693 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 12967
I20260807 11:34:33.630914 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:34:33.631561 24989 external_mini_cluster.cc:1428] Setting key a61e7caeef2c5fb234c32f58e2558ccf
I20260807 11:34:33.634891 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44055
--builtin_ntp_servers=127.24.103.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:34:33.644706 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:33.632063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:33439 (local address 127.24.103.126:44055)
0807 11:34:33.632630 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:33.632636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:33.634020 (+ 1384us) server_negotiation.cc:407] Connection header received
0807 11:34:33.636369 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:33.636379 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:33.636485 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:33.636650 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:33.637840 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:33.639199 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:33.640826 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:33.641653 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:33.641814 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:33.643513 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:33.643522 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:33.643530 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:33.643570 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:33.643630 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:33.643651 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:33.643756 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:33.644063 (+ 307us) server_negotiation.cc:299] Negotiation successful
0807 11:34:33.644196 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":274,"threads_started":1}
I20260807 11:34:33.645789 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:33.632675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44055 (local address 127.24.103.66:33439)
0807 11:34:33.633820 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:33.633870 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:33.636021 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:33.636904 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:33.636914 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:33.637011 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:33.637651 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:33.637669 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:33.639393 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:33.639397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:33.640662 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:33.640674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:33.641931 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:33.641937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:33.642230 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:33.643133 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:34:33.643136 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:33.643190 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:34:33.643287 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:33.643775 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:33.643785 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:33.643792 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 11:34:33.644020 (+ 228us) client_negotiation.cc:240] Negotiation successful
0807 11:34:33.644506 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":364,"threads_started":1}
I20260807 11:34:33.647565 13109 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44055
I20260807 11:34:33.647921 13109 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:33.648730 13109 heartbeater.cc:511] Master 127.24.103.126:44055 requested a full tablet report, sending...
I20260807 11:34:33.650362 12752 ts_manager.cc:194] Registered new tserver with Master: 45132ac8b6c9470fbdc79840c92cb277 (127.24.103.66:35785)
I20260807 11:34:33.651283 12752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.66:33439
W20260807 11:34:34.000608 13113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:34.001044 13113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:34.001195 13113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:34.011497 13113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:34.011651 13113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:34.011821 13113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:34:34.021733 13113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:34.023248 13113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:34.025094 13113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:34.039389 13121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:34.039710 13120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:34.039598 13124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:34.040212 13113 server_base.cc:1034] running on GCE node
I20260807 11:34:34.040829 13113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:34.041814 13113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:34.043072 13113 hybrid_clock.cc:648] HybridClock initialized: now 1786102474042969 us; error 53 us; skew 500 ppm
I20260807 11:34:34.043426 13113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:34.045867 13113 webserver.cc:460] Webserver started at http://127.24.103.67:32913/ using document root <none> and password file <none>
I20260807 11:34:34.046576 13113 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:34.046666 13113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:34.047066 13113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:34.049995 13113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "53034219bb0d4e539fa8019aed0b9a01"
format_stamp: "Formatted at 2026-08-07 11:34:34 on dist-test-slave-wl2h"
server_key: "33404bfe4ccaa7fa6ab4ad0b96755edd"
server_key_iv: "129bde57309f322b63cfd22892ff087e"
server_key_version: "encryptionkey@0"
I20260807 11:34:34.050873 13113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "53034219bb0d4e539fa8019aed0b9a01"
format_stamp: "Formatted at 2026-08-07 11:34:34 on dist-test-slave-wl2h"
server_key: "33404bfe4ccaa7fa6ab4ad0b96755edd"
server_key_iv: "129bde57309f322b63cfd22892ff087e"
server_key_version: "encryptionkey@0"
I20260807 11:34:34.056283 13113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 11:34:34.060020 13136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:34.061491 13113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:34:34.061705 13113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "53034219bb0d4e539fa8019aed0b9a01"
format_stamp: "Formatted at 2026-08-07 11:34:34 on dist-test-slave-wl2h"
server_key: "33404bfe4ccaa7fa6ab4ad0b96755edd"
server_key_iv: "129bde57309f322b63cfd22892ff087e"
server_key_version: "encryptionkey@0"
I20260807 11:34:34.061942 13113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:34.084087 13113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:34.084959 13113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:34.085285 13113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:34.086545 13113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:34.088217 13113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:34.088311 13113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:34.088441 13113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:34.088536 13113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:34.124599 13113 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:32835
I20260807 11:34:34.124698 13264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:32835 every 8 connection(s)
I20260807 11:34:34.126209 13113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:34:34.133522 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 13113
I20260807 11:34:34.133750 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:34:34.134318 24989 external_mini_cluster.cc:1428] Setting key 196a61d466e08dd0409e8721bc5f74f7
I20260807 11:34:34.138867 12961 heartbeater.cc:503] Master 127.24.103.126:44055 was elected leader, sending a full tablet report...
I20260807 11:34:34.143829 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.129836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:54517 (local address 127.24.103.126:44055)
0807 11:34:34.130171 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:34.130180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:34.131673 (+ 1493us) server_negotiation.cc:407] Connection header received
0807 11:34:34.135188 (+ 3515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:34.135194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:34.135295 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:34.135438 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:34.137596 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.138517 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.140786 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.141182 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.141295 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.142827 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:34.142831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:34.142835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:34.142853 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:34.142885 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:34.142896 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:34.142968 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:34.143470 (+ 502us) server_negotiation.cc:299] Negotiation successful
0807 11:34:34.143553 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260807 11:34:34.145325 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.130145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:44055 (local address 127.24.103.67:54517)
0807 11:34:34.131444 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.131499 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:34.134633 (+ 3134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:34.135954 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:34.135976 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:34.136184 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:34.137362 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.137389 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.138709 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.138716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.140599 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.140617 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.141402 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.141408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.141664 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.142481 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:34:34.142484 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:34.142527 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:34:34.142631 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:34.143101 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:34.143118 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:34.143130 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 11:34:34.143447 (+ 317us) client_negotiation.cc:240] Negotiation successful
0807 11:34:34.144010 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":324,"threads_started":1}
I20260807 11:34:34.147079 13267 heartbeater.cc:347] Connected to a master server at 127.24.103.126:44055
I20260807 11:34:34.147434 13267 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:34.148221 13267 heartbeater.cc:511] Master 127.24.103.126:44055 requested a full tablet report, sending...
I20260807 11:34:34.150151 12752 ts_manager.cc:194] Registered new tserver with Master: 53034219bb0d4e539fa8019aed0b9a01 (127.24.103.67:32835)
I20260807 11:34:34.150897 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:34:34.151304 12752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.67:54517
I20260807 11:34:34.162982 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.155907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.24.103.126:44055)
0807 11:34:34.156099 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:34.156104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:34.156632 (+ 528us) server_negotiation.cc:407] Connection header received
0807 11:34:34.156788 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:34.156792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:34.156855 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:34.156936 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:34.157545 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.158416 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.160288 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.160712 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.160814 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.162178 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:34.162184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:34.162187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:34.162209 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:34.162244 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:34.162259 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:34.162347 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:34.162534 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 11:34:34.162620 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 11:34:34.167892 12752 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:34:34.176169 12752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:34:34.217739 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.191049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.24.103.66:35785)
0807 11:34:34.205731 (+ 14682us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:34.205750 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:34.206174 (+ 424us) server_negotiation.cc:407] Connection header received
0807 11:34:34.207163 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:34.207182 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:34.207590 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:34.207913 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:34.209787 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.211889 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.215185 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.216394 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.216560 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.216862 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:34.217164 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 11:34:34.217357 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14511,"thread_start_us":13857,"threads_started":1}
I20260807 11:34:34.217795 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.190812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:35785 (local address 127.0.0.1:45966)
0807 11:34:34.205956 (+ 15144us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.206014 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:34.206247 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:34.208392 (+ 2145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:34.208401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:34.208506 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:34.209453 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.209480 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.212132 (+ 2652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.212138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.214896 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.214912 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.216637 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.216644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.216867 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.216930 (+ 63us) client_negotiation.cc:769] Sending connection context
0807 11:34:34.217106 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 11:34:34.217328 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14830,"mutex_wait_us":1,"spinlock_wait_cycles":65920,"thread_start_us":14647,"threads_started":1}
I20260807 11:34:34.218621 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.191477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.24.103.65:34095)
0807 11:34:34.204437 (+ 12960us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:34.204457 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:34.206645 (+ 2188us) server_negotiation.cc:407] Connection header received
0807 11:34:34.207294 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:34.207307 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:34.207635 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:34.207881 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:34.209660 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.211339 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.214932 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.216491 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.216629 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.216937 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:34.218078 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0807 11:34:34.218251 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12647,"thread_start_us":12991,"threads_started":1}
I20260807 11:34:34.219373 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.190964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34095 (local address 127.0.0.1:60752)
0807 11:34:34.206512 (+ 15548us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.206532 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:34.206796 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:34.208387 (+ 1591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:34.208398 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:34.208446 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:34.209450 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.209467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.211499 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.211506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.214655 (+ 3149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.214680 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.217083 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.217090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.217276 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.217308 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 11:34:34.217438 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 11:34:34.218923 (+ 1485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15402,"mutex_wait_us":24,"thread_start_us":14484,"threads_started":1}
I20260807 11:34:34.219885 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.206657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:32835 (local address 127.0.0.1:38390)
0807 11:34:34.209036 (+ 2379us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.209069 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:34.209863 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:34.211884 (+ 2021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:34.211888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:34.211916 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:34.212593 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.212603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.214552 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.214556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.216732 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.216748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.219002 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.219027 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.219223 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.219272 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 11:34:34.219375 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 11:34:34.219520 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2037,"mutex_wait_us":134,"thread_start_us":2324,"threads_started":1}
I20260807 11:34:34.220243 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.206774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.24.103.67:32835)
0807 11:34:34.211279 (+ 4505us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:34.211290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:34.211327 (+ 37us) server_negotiation.cc:407] Connection header received
0807 11:34:34.211439 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:34.211448 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:34.211674 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:34.211896 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:34.212900 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.214371 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.216975 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.218431 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.218571 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.218865 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:34.219644 (+ 779us) server_negotiation.cc:299] Negotiation successful
0807 11:34:34.219842 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4331,"thread_start_us":4144,"threads_started":1}
I20260807 11:34:34.222896 12888 tablet_service.cc:1467] Processing CreateTablet for tablet 98af5c110ec54c9d990887fd3cd66da4 (DEFAULT_TABLE table=test-table [id=3db30cd6b0de4ba79974240582d8e602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:34.222888 13039 tablet_service.cc:1467] Processing CreateTablet for tablet 98af5c110ec54c9d990887fd3cd66da4 (DEFAULT_TABLE table=test-table [id=3db30cd6b0de4ba79974240582d8e602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:34.224480 13039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98af5c110ec54c9d990887fd3cd66da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:34.224002 13193 tablet_service.cc:1467] Processing CreateTablet for tablet 98af5c110ec54c9d990887fd3cd66da4 (DEFAULT_TABLE table=test-table [id=3db30cd6b0de4ba79974240582d8e602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:34.224812 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98af5c110ec54c9d990887fd3cd66da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:34.225507 13193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98af5c110ec54c9d990887fd3cd66da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:34.241130 13289 tablet_bootstrap.cc:492] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: Bootstrap starting.
I20260807 11:34:34.244024 13288 tablet_bootstrap.cc:492] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0: Bootstrap starting.
I20260807 11:34:34.245277 13291 tablet_bootstrap.cc:492] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: Bootstrap starting.
I20260807 11:34:34.246327 13289 tablet_bootstrap.cc:654] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:34.247946 13288 tablet_bootstrap.cc:654] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:34.247983 13289 log.cc:824] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:34.248880 13291 tablet_bootstrap.cc:654] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:34.249225 13288 log.cc:824] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:34.249948 13291 log.cc:824] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:34.251472 13289 tablet_bootstrap.cc:492] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: No bootstrap required, opened a new log
I20260807 11:34:34.251752 13289 ts_tablet_manager.cc:1397] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260807 11:34:34.252315 13288 tablet_bootstrap.cc:492] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0: No bootstrap required, opened a new log
I20260807 11:34:34.252597 13288 ts_tablet_manager.cc:1397] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 11:34:34.253590 13291 tablet_bootstrap.cc:492] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: No bootstrap required, opened a new log
I20260807 11:34:34.253845 13291 ts_tablet_manager.cc:1397] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:34:34.261081 13288 raft_consensus.cc:348] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34095 } } peers { permanent_uuid: "45132ac8b6c9470fbdc79840c92cb277" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35785 } } peers { permanent_uuid: "53034219bb0d4e539fa8019aed0b9a01" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 32835 } }
I20260807 11:34:34.261498 13288 raft_consensus.cc:374] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:34.261615 13288 raft_consensus.cc:729] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1373c99fc82495fbb3125bcfdeff5f0, State: Initialized, Role: FOLLOWER
I20260807 11:34:34.262367 13288 consensus_queue.cc:260] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34095 } } peers { permanent_uuid: "45132ac8b6c9470fbdc79840c92cb277" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35785 } } peers { permanent_uuid: "53034219bb0d4e539fa8019aed0b9a01" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 32835 } }
I20260807 11:34:34.262583 13289 raft_consensus.cc:348] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34095 } } peers { permanent_uuid: "45132ac8b6c9470fbdc79840c92cb277" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35785 } } peers { permanent_uuid: "53034219bb0d4e539fa8019aed0b9a01" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 32835 } }
I20260807 11:34:34.263047 13289 raft_consensus.cc:374] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:34.263226 13289 raft_consensus.cc:729] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45132ac8b6c9470fbdc79840c92cb277, State: Initialized, Role: FOLLOWER
I20260807 11:34:34.263926 13289 consensus_queue.cc:260] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34095 } } peers { permanent_uuid: "45132ac8b6c9470fbdc79840c92cb277" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35785 } } peers { permanent_uuid: "53034219bb0d4e539fa8019aed0b9a01" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 32835 } }
I20260807 11:34:34.264521 13291 raft_consensus.cc:348] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34095 } } peers { permanent_uuid: "45132ac8b6c9470fbdc79840c92cb277" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35785 } } peers { permanent_uuid: "53034219bb0d4e539fa8019aed0b9a01" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 32835 } }
I20260807 11:34:34.265157 13109 heartbeater.cc:503] Master 127.24.103.126:44055 was elected leader, sending a full tablet report...
I20260807 11:34:34.265767 13289 ts_tablet_manager.cc:1428] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260807 11:34:34.266172 13288 ts_tablet_manager.cc:1428] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260807 11:34:34.266234 13291 raft_consensus.cc:374] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:34.266356 13291 raft_consensus.cc:729] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53034219bb0d4e539fa8019aed0b9a01, State: Initialized, Role: FOLLOWER
I20260807 11:34:34.266885 13291 consensus_queue.cc:260] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34095 } } peers { permanent_uuid: "45132ac8b6c9470fbdc79840c92cb277" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35785 } } peers { permanent_uuid: "53034219bb0d4e539fa8019aed0b9a01" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 32835 } }
I20260807 11:34:34.267905 13267 heartbeater.cc:503] Master 127.24.103.126:44055 was elected leader, sending a full tablet report...
I20260807 11:34:34.270453 13291 ts_tablet_manager.cc:1428] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260807 11:34:34.363549 12962 tablet.cc:2369] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:34:34.378177 13268 tablet.cc:2369] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:34:34.381273 13110 tablet.cc:2369] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:34:34.433076 13296 raft_consensus.cc:482] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:34:34.433476 13296 raft_consensus.cc:504] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34095 } } peers { permanent_uuid: "45132ac8b6c9470fbdc79840c92cb277" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35785 } } peers { permanent_uuid: "53034219bb0d4e539fa8019aed0b9a01" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 32835 } }
I20260807 11:34:34.435343 13296 leader_election.cc:288] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45132ac8b6c9470fbdc79840c92cb277 (127.24.103.66:35785), 53034219bb0d4e539fa8019aed0b9a01 (127.24.103.67:32835)
I20260807 11:34:34.442868 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.437146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:51513 (local address 127.24.103.67:32835)
0807 11:34:34.437358 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:34.437364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:34.437574 (+ 210us) server_negotiation.cc:407] Connection header received
0807 11:34:34.437790 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:34.437794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:34.437852 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:34.437953 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:34.438515 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.439314 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.441141 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.441912 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.442105 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.442175 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:34.442484 (+ 309us) server_negotiation.cc:299] Negotiation successful
0807 11:34:34.442614 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 11:34:34.442986 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.435979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:45565 (local address 127.24.103.66:35785)
0807 11:34:34.436176 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:34.436181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:34.436942 (+ 761us) server_negotiation.cc:407] Connection header received
0807 11:34:34.437238 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:34.437244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:34.437345 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:34.437419 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:34.438162 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.438927 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.441079 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.441923 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.442114 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.442246 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:34.442572 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 11:34:34.442712 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 11:34:34.443111 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.435877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:35785 (local address 127.24.103.65:45565)
0807 11:34:34.436794 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.436815 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:34.436994 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:34.437649 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:34.437653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:34.437684 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:34.437992 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.438000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.439133 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.439137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.440871 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.440887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.442202 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.442210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.442387 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.442413 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 11:34:34.442542 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 11:34:34.442693 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":24,"thread_start_us":539,"threads_started":1}
I20260807 11:34:34.443396 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.436886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:32835 (local address 127.24.103.65:51513)
0807 11:34:34.437466 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.437481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:34.437638 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:34.437980 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:34.437984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:34.438007 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:34.438367 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.438376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.439459 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.439464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.440931 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:34.440944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.442123 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.442130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:34.442319 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.442344 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 11:34:34.442445 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 11:34:34.442581 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":5,"thread_start_us":472,"threads_started":1}
I20260807 11:34:34.444489 13061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af5c110ec54c9d990887fd3cd66da4" candidate_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45132ac8b6c9470fbdc79840c92cb277" is_pre_election: true
I20260807 11:34:34.444772 13214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af5c110ec54c9d990887fd3cd66da4" candidate_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53034219bb0d4e539fa8019aed0b9a01" is_pre_election: true
I20260807 11:34:34.445129 13061 raft_consensus.cc:2454] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1373c99fc82495fbb3125bcfdeff5f0 in term 0.
I20260807 11:34:34.445326 13214 raft_consensus.cc:2454] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1373c99fc82495fbb3125bcfdeff5f0 in term 0.
I20260807 11:34:34.445901 12839 leader_election.cc:302] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45132ac8b6c9470fbdc79840c92cb277, d1373c99fc82495fbb3125bcfdeff5f0; no voters:
I20260807 11:34:34.446290 13296 raft_consensus.cc:2781] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:34:34.446487 13296 raft_consensus.cc:482] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:34:34.446626 13296 raft_consensus.cc:3037] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:34.449077 13296 raft_consensus.cc:504] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34095 } } peers { permanent_uuid: "45132ac8b6c9470fbdc79840c92cb277" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35785 } } peers { permanent_uuid: "53034219bb0d4e539fa8019aed0b9a01" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 32835 } }
I20260807 11:34:34.449728 13296 leader_election.cc:288] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [CANDIDATE]: Term 1 election: Requested vote from peers 45132ac8b6c9470fbdc79840c92cb277 (127.24.103.66:35785), 53034219bb0d4e539fa8019aed0b9a01 (127.24.103.67:32835)
I20260807 11:34:34.450251 13061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af5c110ec54c9d990887fd3cd66da4" candidate_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45132ac8b6c9470fbdc79840c92cb277"
I20260807 11:34:34.450428 13214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af5c110ec54c9d990887fd3cd66da4" candidate_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53034219bb0d4e539fa8019aed0b9a01"
I20260807 11:34:34.450479 13061 raft_consensus.cc:3037] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:34.450645 13214 raft_consensus.cc:3037] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:34.453840 13214 raft_consensus.cc:2454] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1373c99fc82495fbb3125bcfdeff5f0 in term 1.
I20260807 11:34:34.453860 13061 raft_consensus.cc:2454] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1373c99fc82495fbb3125bcfdeff5f0 in term 1.
I20260807 11:34:34.454228 12839 leader_election.cc:302] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53034219bb0d4e539fa8019aed0b9a01, d1373c99fc82495fbb3125bcfdeff5f0; no voters:
I20260807 11:34:34.454505 13296 raft_consensus.cc:2781] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:34.455049 13296 raft_consensus.cc:686] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 1 LEADER]: Becoming Leader. State: Replica: d1373c99fc82495fbb3125bcfdeff5f0, State: Running, Role: LEADER
I20260807 11:34:34.455591 13296 consensus_queue.cc:237] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34095 } } peers { permanent_uuid: "45132ac8b6c9470fbdc79840c92cb277" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35785 } } peers { permanent_uuid: "53034219bb0d4e539fa8019aed0b9a01" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 32835 } }
I20260807 11:34:34.460623 12752 catalog_manager.cc:5495] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1373c99fc82495fbb3125bcfdeff5f0 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1373c99fc82495fbb3125bcfdeff5f0" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45132ac8b6c9470fbdc79840c92cb277" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53034219bb0d4e539fa8019aed0b9a01" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 32835 } health_report { overall_health: UNKNOWN } } }
I20260807 11:34:34.514351 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.508979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.24.103.65:34095)
0807 11:34:34.509183 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:34.509187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:34.509205 (+ 18us) server_negotiation.cc:407] Connection header received
0807 11:34:34.509402 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:34.509406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:34.509501 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:34.509629 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:34:34.510007 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.510720 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.512204 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.512645 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.512752 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.513291 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:34:34.513835 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:34:34.513952 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:34.514071 (+ 119us) server_negotiation.cc:299] Negotiation successful
0807 11:34:34.514145 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 11:34:34.522742 13214 raft_consensus.cc:1261] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01 [term 1 FOLLOWER]: Refusing update from remote peer d1373c99fc82495fbb3125bcfdeff5f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:34.522742 13061 raft_consensus.cc:1261] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277 [term 1 FOLLOWER]: Refusing update from remote peer d1373c99fc82495fbb3125bcfdeff5f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:34.523779 13302 consensus_queue.cc:1035] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45132ac8b6c9470fbdc79840c92cb277" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 35785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:34:34.524169 13296 consensus_queue.cc:1035] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53034219bb0d4e539fa8019aed0b9a01" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:34:34.537297 13306 mvcc.cc:205] Tried to move back new op lower bound from 7315875735635611648 to 7315875735373643776. Current Snapshot: MvccSnapshot[applied={T|T < 7315875735635611648}]
I20260807 11:34:34.552940 12752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:34.553233 12752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:34.557432 12752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6a9caa28efd4e66a18ebe844c4defd3: Sending DeleteTablet for 3 replicas of tablet 98af5c110ec54c9d990887fd3cd66da4
I20260807 11:34:34.558859 12888 tablet_service.cc:1514] Processing DeleteTablet for tablet 98af5c110ec54c9d990887fd3cd66da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:34 UTC) from {username='slave'} at 127.0.0.1:60752
I20260807 11:34:34.559111 13039 tablet_service.cc:1514] Processing DeleteTablet for tablet 98af5c110ec54c9d990887fd3cd66da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:34 UTC) from {username='slave'} at 127.0.0.1:45966
I20260807 11:34:34.559325 13193 tablet_service.cc:1514] Processing DeleteTablet for tablet 98af5c110ec54c9d990887fd3cd66da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:34 UTC) from {username='slave'} at 127.0.0.1:38390
I20260807 11:34:34.561369 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 12816
I20260807 11:34:34.569573 13317 tablet_replica.cc:330] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: stopping tablet replica
I20260807 11:34:34.570186 13317 raft_consensus.cc:2229] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:34.570861 13317 raft_consensus.cc:2258] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:34.573031 13316 tablet_replica.cc:330] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0: stopping tablet replica
I20260807 11:34:34.573822 13316 raft_consensus.cc:2229] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:34:34.574546 13319 tablet_replica.cc:330] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: stopping tablet replica
I20260807 11:34:34.574697 13316 raft_consensus.cc:2258] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:34.574983 13319 raft_consensus.cc:2229] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:34.576352 13319 raft_consensus.cc:2258] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:34.577701 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.563301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.24.103.65:34095)
0807 11:34:34.563531 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:34.563538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:34.567935 (+ 4397us) server_negotiation.cc:407] Connection header received
0807 11:34:34.568172 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:34.568178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:34.568290 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:34.568400 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:34.569892 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.571203 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.573267 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.574020 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.574127 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.575836 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:34.575848 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:34.575866 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:34.576713 (+ 847us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:34.576787 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:34.576813 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:34.576899 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:34.577199 (+ 300us) server_negotiation.cc:299] Negotiation successful
0807 11:34:34.577303 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:34:34.578181 13316 ts_tablet_manager.cc:1905] T 98af5c110ec54c9d990887fd3cd66da4 P d1373c99fc82495fbb3125bcfdeff5f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:34.578514 12955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:34.579120 13319 ts_tablet_manager.cc:1905] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:34.580765 13317 ts_tablet_manager.cc:1905] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:34.586138 13317 ts_tablet_manager.cc:1918] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:34.586342 13317 log.cc:1196] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/98af5c110ec54c9d990887fd3cd66da4
I20260807 11:34:34.586833 13317 ts_tablet_manager.cc:1939] T 98af5c110ec54c9d990887fd3cd66da4 P 45132ac8b6c9470fbdc79840c92cb277: Deleting consensus metadata
I20260807 11:34:34.587584 13319 ts_tablet_manager.cc:1918] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:34.587751 13319 log.cc:1196] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/98af5c110ec54c9d990887fd3cd66da4
I20260807 11:34:34.588304 13319 ts_tablet_manager.cc:1939] T 98af5c110ec54c9d990887fd3cd66da4 P 53034219bb0d4e539fa8019aed0b9a01: Deleting consensus metadata
I20260807 11:34:34.588642 12737 catalog_manager.cc:4841] TS 45132ac8b6c9470fbdc79840c92cb277 (127.24.103.66:35785): tablet 98af5c110ec54c9d990887fd3cd66da4 (table test-table [id=3db30cd6b0de4ba79974240582d8e602]) successfully deleted
I20260807 11:34:34.590133 12737 catalog_manager.cc:4841] TS 53034219bb0d4e539fa8019aed0b9a01 (127.24.103.67:32835): tablet 98af5c110ec54c9d990887fd3cd66da4 (table test-table [id=3db30cd6b0de4ba79974240582d8e602]) successfully deleted
I20260807 11:34:34.682036 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 12816
I20260807 11:34:34.703882 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 12967
W20260807 11:34:34.703841 12736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:34095 (error 108)
I20260807 11:34:34.705826 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.705193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34095 (local address 127.0.0.1:60790)
0807 11:34:34.705440 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.705628 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260807 11:34:34.706096 12736 catalog_manager.cc:4584] TS d1373c99fc82495fbb3125bcfdeff5f0 (127.24.103.65:34095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98af5c110ec54c9d990887fd3cd66da4: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34095: connect: Connection refused (error 111)
I20260807 11:34:34.711609 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.704698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.24.103.66:35785)
0807 11:34:34.704930 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:34.704938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:34.705061 (+ 123us) server_negotiation.cc:407] Connection header received
0807 11:34:34.705387 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:34.705394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:34.705512 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:34.705644 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:34.706129 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.706963 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.708432 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.708869 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.708967 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.710020 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:34.710028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:34.710042 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:34.710684 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:34.710732 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:34.710752 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:34.710844 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:34.711106 (+ 262us) server_negotiation.cc:299] Negotiation successful
0807 11:34:34.711238 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 11:34:34.712571 13103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:34.760075 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.759531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34095 (local address 127.0.0.1:60802)
0807 11:34:34.759788 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.759888 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260807 11:34:34.761302 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.760922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34095 (local address 127.0.0.1:60808)
0807 11:34:34.761111 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.761185 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 11:34:34.806082 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 12967
I20260807 11:34:34.822002 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.821430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34095 (local address 127.0.0.1:60820)
0807 11:34:34.821696 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.821802 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260807 11:34:34.823266 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.822837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34095 (local address 127.0.0.1:60834)
0807 11:34:34.823003 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.823140 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 11:34:34.828001 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 13113
I20260807 11:34:34.835394 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.829006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.24.103.67:32835)
0807 11:34:34.829224 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:34.829230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:34.829252 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:34:34.829415 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:34.829420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:34.829518 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:34.829642 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:34.830047 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.830804 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.832211 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:34.832605 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:34.832757 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:34.833878 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:34.833888 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:34.833903 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:34.834539 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:34.834588 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:34.834608 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:34.834745 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:34.834882 (+ 137us) server_negotiation.cc:299] Negotiation successful
0807 11:34:34.834991 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:34:34.836110 13260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:34.904220 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.903620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34095 (local address 127.0.0.1:60840)
0807 11:34:34.903893 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.904032 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260807 11:34:34.905536 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:34.905088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34095 (local address 127.0.0.1:60848)
0807 11:34:34.905335 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:34.905418 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260807 11:34:34.925686 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 13113
I20260807 11:34:34.946261 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 12718
I20260807 11:34:34.947444 12786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:35.061462 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 12718
2026-08-07T11:34:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3138 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:34:35 dist-test-slave-wl2h krb5kdc[13328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:34:35 dist-test-slave-wl2h krb5kdc[13328](info): set up 2 sockets
Aug 07 11:34:35 dist-test-slave-wl2h krb5kdc[13328](info): commencing operation
krb5kdc: starting...
W20260807 11:34:36.296069 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:34:36 dist-test-slave-wl2h krb5kdc[13328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:34:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:34:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:34:36.634755 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41503
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:41503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:36.999727 13344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:37.000113 13344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:37.000279 13344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:37.008147 13344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:34:37.008320 13344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:37.008424 13344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:37.008536 13344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:34:37.008627 13344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:34:37.019933 13344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:41503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:37.021500 13344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:37.022915 13344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:37.036594 13349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:37.036594 13354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:37.036594 13350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:37.037982 13344 server_base.cc:1034] running on GCE node
I20260807 11:34:37.038952 13344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:37.040628 13344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:37.041899 13344 hybrid_clock.cc:648] HybridClock initialized: now 1786102477041841 us; error 74 us; skew 500 ppm
I20260807 11:34:37.042289 13344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:37 dist-test-slave-wl2h krb5kdc[13328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102477, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:37.045869 13344 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:34:37.047574 13344 webserver.cc:460] Webserver started at http://127.24.103.126:42849/ using document root <none> and password file <none>
I20260807 11:34:37.048354 13344 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:37.048516 13344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:37.048853 13344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:37.052206 13344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "3ba3e83c71be45039b3e9ef17dc1df28"
format_stamp: "Formatted at 2026-08-07 11:34:37 on dist-test-slave-wl2h"
server_key: "e28aea50ffe1fbbdd35e258ddc944c5f"
server_key_iv: "66714a475ceb52335cd165c901d14972"
server_key_version: "encryptionkey@0"
I20260807 11:34:37.053025 13344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "3ba3e83c71be45039b3e9ef17dc1df28"
format_stamp: "Formatted at 2026-08-07 11:34:37 on dist-test-slave-wl2h"
server_key: "e28aea50ffe1fbbdd35e258ddc944c5f"
server_key_iv: "66714a475ceb52335cd165c901d14972"
server_key_version: "encryptionkey@0"
I20260807 11:34:37.059151 13344 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260807 11:34:37.062872 13362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:37.064494 13344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 11:34:37.064724 13344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "3ba3e83c71be45039b3e9ef17dc1df28"
format_stamp: "Formatted at 2026-08-07 11:34:37 on dist-test-slave-wl2h"
server_key: "e28aea50ffe1fbbdd35e258ddc944c5f"
server_key_iv: "66714a475ceb52335cd165c901d14972"
server_key_version: "encryptionkey@0"
I20260807 11:34:37.064990 13344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:37.092497 13344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:37.097254 13344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:37.097599 13344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:37.119699 13344 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:41503
I20260807 11:34:37.119742 13426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:41503 every 8 connection(s)
I20260807 11:34:37.121260 13344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:34:37.125399 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:37.130512 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 13344
I20260807 11:34:37.130759 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:34:37.131376 24989 external_mini_cluster.cc:1428] Setting key c8a0c07ad5cbd197f9740fa7f6be6675
I20260807 11:34:37.138103 13427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28: Bootstrap starting.
I20260807 11:34:37.142086 13427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:37.143240 13427 log.cc:824] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:37.146207 13427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28: No bootstrap required, opened a new log
Aug 07 11:34:37 dist-test-slave-wl2h krb5kdc[13328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:37.154402 13427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba3e83c71be45039b3e9ef17dc1df28" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41503 } }
I20260807 11:34:37.154881 13427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:37.155001 13427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba3e83c71be45039b3e9ef17dc1df28, State: Initialized, Role: FOLLOWER
I20260807 11:34:37.155995 13427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba3e83c71be45039b3e9ef17dc1df28" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41503 } }
I20260807 11:34:37.156541 13427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:34:37.156735 13427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:34:37.156932 13427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:37.159307 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:37.134238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.24.103.126:41503)
0807 11:34:37.135525 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:37.135539 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:37.135589 (+ 50us) server_negotiation.cc:407] Connection header received
0807 11:34:37.137595 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:37.137625 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:37.138113 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:37.138550 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:37.140640 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:37.142102 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:37.143798 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:37.144595 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:37.144815 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:37.149680 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:37.149689 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:37.149705 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:37.149760 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:37.153629 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:37.154561 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:37.154568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:37.154576 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:37.154770 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:37.155408 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:37.155415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:37.155418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:37.156441 (+ 1023us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:37.156802 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:37.157460 (+ 658us) server_negotiation.cc:299] Negotiation successful
0807 11:34:37.158012 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":546,"threads_started":1}
I20260807 11:34:37.160496 13427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba3e83c71be45039b3e9ef17dc1df28" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41503 } }
I20260807 11:34:37.161149 13427 leader_election.cc:302] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ba3e83c71be45039b3e9ef17dc1df28; no voters:
I20260807 11:34:37.161932 13427 leader_election.cc:288] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:34:37.162112 13434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:37.163843 13434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba3e83c71be45039b3e9ef17dc1df28, State: Running, Role: LEADER
I20260807 11:34:37.164498 13434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba3e83c71be45039b3e9ef17dc1df28" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41503 } }
I20260807 11:34:37.164752 13427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:34:37.168341 13435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ba3e83c71be45039b3e9ef17dc1df28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba3e83c71be45039b3e9ef17dc1df28" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41503 } } }
I20260807 11:34:37.168330 13436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba3e83c71be45039b3e9ef17dc1df28. Latest consensus state: current_term: 1 leader_uuid: "3ba3e83c71be45039b3e9ef17dc1df28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba3e83c71be45039b3e9ef17dc1df28" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41503 } } }
I20260807 11:34:37.168956 13436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:37.168970 13435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:37.169992 13446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:34:37.177047 13446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:34:37.188180 13446 catalog_manager.cc:1347] Generated new cluster ID: 78cbcb1bfe9645108bc3b0ec79d7d2a9
I20260807 11:34:37.188352 13446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:34:37.197245 13446 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:34:37.199621 13446 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:34:37.209115 13446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28: Generated new TSK 0
I20260807 11:34:37.210085 13446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:34:37.306957 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41503
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:37.671838 13462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:37.672276 13462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:37.672479 13462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:37.681126 13462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:37.681236 13462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:37.681434 13462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:34:37.691448 13462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:37.692859 13462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:37.694231 13462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:37.708325 13472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:37.708477 13469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:37.708487 13468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:37.709086 13462 server_base.cc:1034] running on GCE node
I20260807 11:34:37.709828 13462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:37.710892 13462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:37.712158 13462 hybrid_clock.cc:648] HybridClock initialized: now 1786102477712128 us; error 65 us; skew 500 ppm
I20260807 11:34:37.712545 13462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:37 dist-test-slave-wl2h krb5kdc[13328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102477, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:37.716054 13462 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:34:37.717861 13462 webserver.cc:460] Webserver started at http://127.24.103.65:41843/ using document root <none> and password file <none>
I20260807 11:34:37.718631 13462 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:37.718789 13462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:37.719195 13462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:37.722210 13462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "e79860f789304a3ba16bcd4a8a4acffe"
format_stamp: "Formatted at 2026-08-07 11:34:37 on dist-test-slave-wl2h"
server_key: "d6ec1d383146a2e7f831fbb77f21de26"
server_key_iv: "8d3ff088401170e6b7d6194c2c38fe1e"
server_key_version: "encryptionkey@0"
I20260807 11:34:37.722983 13462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "e79860f789304a3ba16bcd4a8a4acffe"
format_stamp: "Formatted at 2026-08-07 11:34:37 on dist-test-slave-wl2h"
server_key: "d6ec1d383146a2e7f831fbb77f21de26"
server_key_iv: "8d3ff088401170e6b7d6194c2c38fe1e"
server_key_version: "encryptionkey@0"
I20260807 11:34:37.728408 13462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 11:34:37.732196 13479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:37.733719 13462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:34:37.733914 13462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "e79860f789304a3ba16bcd4a8a4acffe"
format_stamp: "Formatted at 2026-08-07 11:34:37 on dist-test-slave-wl2h"
server_key: "d6ec1d383146a2e7f831fbb77f21de26"
server_key_iv: "8d3ff088401170e6b7d6194c2c38fe1e"
server_key_version: "encryptionkey@0"
I20260807 11:34:37.734141 13462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:37.756803 13462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:37.761288 13462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:37.761620 13462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:37.762681 13462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:37.764268 13462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:37.764379 13462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:37.764557 13462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:37.764612 13462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:37.802238 13462 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:34535
I20260807 11:34:37.802385 13610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:34535 every 8 connection(s)
I20260807 11:34:37.803805 13462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:34:37.810725 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 13462
I20260807 11:34:37.810953 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:34:37.811601 24989 external_mini_cluster.cc:1428] Setting key fcc637121b6c88cdd21bd19d550bf40c
Aug 07 11:34:37 dist-test-slave-wl2h krb5kdc[13328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102477, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:37.829676 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:37.807203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:39407 (local address 127.24.103.126:41503)
0807 11:34:37.807729 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:37.807735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:37.808678 (+ 943us) server_negotiation.cc:407] Connection header received
0807 11:34:37.811114 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:37.811121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:37.811214 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:37.811332 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:37.813472 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:37.814509 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:37.817309 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:37.817822 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:37.817957 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:37.822812 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:37.822817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:37.822821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:37.822860 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:37.825908 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:37.826808 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:37.826813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:37.826816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:37.826947 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:37.827525 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:37.827533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:37.827542 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:37.828006 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:37.828196 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:37.829170 (+ 974us) server_negotiation.cc:299] Negotiation successful
0807 11:34:37.829370 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":275,"threads_started":1}
I20260807 11:34:37.830677 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:37.807694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41503 (local address 127.24.103.65:39407)
0807 11:34:37.808493 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:37.808534 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:37.810633 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:37.811630 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:37.811644 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:37.812555 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:37.813272 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:37.813290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:37.814727 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:37.814735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:37.817109 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:37.817129 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:37.818094 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:37.818104 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:37.818468 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:37.819425 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:34:37.819463 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:37.822499 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:37.826097 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:37.826108 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:37.826128 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:37.826641 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:37.827145 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:37.827151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:37.827155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:37.827382 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:37.828189 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:37.828195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:37.828714 (+ 519us) client_negotiation.cc:769] Sending connection context
0807 11:34:37.828981 (+ 267us) client_negotiation.cc:240] Negotiation successful
0807 11:34:37.829466 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":316,"threads_started":1}
I20260807 11:34:37.832449 13611 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41503
I20260807 11:34:37.832875 13611 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:37.833954 13611 heartbeater.cc:511] Master 127.24.103.126:41503 requested a full tablet report, sending...
I20260807 11:34:37.836438 13381 ts_manager.cc:194] Registered new tserver with Master: e79860f789304a3ba16bcd4a8a4acffe (127.24.103.65:34535)
I20260807 11:34:37.838024 13381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:39407
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:34:37.887848 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41503
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:38.243153 13617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:38.243427 13617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:38.243628 13617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:38.251057 13617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:38.251209 13617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:38.251399 13617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:34:38.261198 13617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:38.262558 13617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:38.263983 13617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:38.277941 13626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:38.277995 13624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:38.278017 13622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:38.278923 13617 server_base.cc:1034] running on GCE node
I20260807 11:34:38.279598 13617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:38.280647 13617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:38.281879 13617 hybrid_clock.cc:648] HybridClock initialized: now 1786102478281830 us; error 36 us; skew 500 ppm
I20260807 11:34:38.282256 13617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:38 dist-test-slave-wl2h krb5kdc[13328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102478, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:38.286127 13617 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:34:38.287842 13617 webserver.cc:460] Webserver started at http://127.24.103.66:33213/ using document root <none> and password file <none>
I20260807 11:34:38.288578 13617 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:38.288743 13617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:38.289081 13617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:38.292382 13617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "84d9a986d6a1426696dca538566c817a"
format_stamp: "Formatted at 2026-08-07 11:34:38 on dist-test-slave-wl2h"
server_key: "6c3c0cd0be604e37e2c05765ece39854"
server_key_iv: "73a1c927b54852574a6938ad5808b656"
server_key_version: "encryptionkey@0"
I20260807 11:34:38.293224 13617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "84d9a986d6a1426696dca538566c817a"
format_stamp: "Formatted at 2026-08-07 11:34:38 on dist-test-slave-wl2h"
server_key: "6c3c0cd0be604e37e2c05765ece39854"
server_key_iv: "73a1c927b54852574a6938ad5808b656"
server_key_version: "encryptionkey@0"
I20260807 11:34:38.298794 13617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:34:38.303002 13634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:38.304602 13617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:34:38.304829 13617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "84d9a986d6a1426696dca538566c817a"
format_stamp: "Formatted at 2026-08-07 11:34:38 on dist-test-slave-wl2h"
server_key: "6c3c0cd0be604e37e2c05765ece39854"
server_key_iv: "73a1c927b54852574a6938ad5808b656"
server_key_version: "encryptionkey@0"
I20260807 11:34:38.305073 13617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:38.329916 13617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:38.334775 13617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:38.335180 13617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:38.336274 13617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:38.337783 13617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:38.337908 13617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:38.338066 13617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:38.338140 13617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:38.373368 13617 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:37827
I20260807 11:34:38.373426 13765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:37827 every 8 connection(s)
I20260807 11:34:38.374723 13617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:34:38.383351 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 13617
I20260807 11:34:38.383540 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:34:38.384091 24989 external_mini_cluster.cc:1428] Setting key 461626fa944a641dc8ea7d4fc6c9b27e
Aug 07 11:34:38 dist-test-slave-wl2h krb5kdc[13328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102478, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:38.398679 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:38.377577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:54267 (local address 127.24.103.126:41503)
0807 11:34:38.378093 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:38.378097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:38.378930 (+ 833us) server_negotiation.cc:407] Connection header received
0807 11:34:38.380818 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:38.380824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:38.380916 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:38.381039 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:38.382636 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:38.383717 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:38.385870 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:38.386466 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:38.386621 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:38.390475 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:38.390485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:38.390494 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:38.390545 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:38.394815 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:38.395794 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:38.395803 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:38.395810 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:38.395993 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:38.396497 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:38.396505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:38.396511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:38.396962 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:38.397156 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:38.397843 (+ 687us) server_negotiation.cc:299] Negotiation successful
0807 11:34:38.398130 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":282,"threads_started":1}
I20260807 11:34:38.399401 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:38.378021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41503 (local address 127.24.103.66:54267)
0807 11:34:38.378753 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:38.378801 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:38.380553 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:38.381279 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:38.381287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:38.381806 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:38.382451 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:38.382474 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:38.383874 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:38.383879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:38.385674 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:38.385689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:38.386708 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:38.386714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:38.386919 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:38.387694 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:34:38.387717 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:38.390212 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:38.394995 (+ 4783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:38.395004 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:38.395092 (+ 88us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:38.395651 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:38.396146 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:38.396150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:38.396152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:38.396360 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:38.397093 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:38.397098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:38.397443 (+ 345us) client_negotiation.cc:769] Sending connection context
0807 11:34:38.397679 (+ 236us) client_negotiation.cc:240] Negotiation successful
0807 11:34:38.398110 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":303,"threads_started":1}
I20260807 11:34:38.401336 13766 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41503
I20260807 11:34:38.401750 13766 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:38.402781 13766 heartbeater.cc:511] Master 127.24.103.126:41503 requested a full tablet report, sending...
I20260807 11:34:38.404533 13381 ts_manager.cc:194] Registered new tserver with Master: 84d9a986d6a1426696dca538566c817a (127.24.103.66:37827)
I20260807 11:34:38.405866 13381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:54267
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:34:38.459416 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41503
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:38.810374 13772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:38.810652 13772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:38.810752 13772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:38.818228 13772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:38.818341 13772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:38.818459 13772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:34:38.828426 13772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:41503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:38.829721 13772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:38.830964 13772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 11:34:38.841147 13611 heartbeater.cc:503] Master 127.24.103.126:41503 was elected leader, sending a full tablet report...
W20260807 11:34:38.845176 13778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:38.845176 13779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:38.845216 13782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:38.845804 13772 server_base.cc:1034] running on GCE node
I20260807 11:34:38.846598 13772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:38.847680 13772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:38.848955 13772 hybrid_clock.cc:648] HybridClock initialized: now 1786102478848903 us; error 40 us; skew 500 ppm
I20260807 11:34:38.849347 13772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:38 dist-test-slave-wl2h krb5kdc[13328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102478, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:38.852996 13772 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:34:38.854903 13772 webserver.cc:460] Webserver started at http://127.24.103.67:46335/ using document root <none> and password file <none>
I20260807 11:34:38.855702 13772 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:38.855873 13772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:38.856240 13772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:38.859546 13772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "74185555974c4fe6a9b4b360bb62e547"
format_stamp: "Formatted at 2026-08-07 11:34:38 on dist-test-slave-wl2h"
server_key: "459b05bbda1288478143e96bc601382c"
server_key_iv: "38d97c3e183b59c6a046c28a4be8f1f7"
server_key_version: "encryptionkey@0"
I20260807 11:34:38.860424 13772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "74185555974c4fe6a9b4b360bb62e547"
format_stamp: "Formatted at 2026-08-07 11:34:38 on dist-test-slave-wl2h"
server_key: "459b05bbda1288478143e96bc601382c"
server_key_iv: "38d97c3e183b59c6a046c28a4be8f1f7"
server_key_version: "encryptionkey@0"
I20260807 11:34:38.866478 13772 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 11:34:38.870522 13791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:38.872231 13772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:34:38.872426 13772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "74185555974c4fe6a9b4b360bb62e547"
format_stamp: "Formatted at 2026-08-07 11:34:38 on dist-test-slave-wl2h"
server_key: "459b05bbda1288478143e96bc601382c"
server_key_iv: "38d97c3e183b59c6a046c28a4be8f1f7"
server_key_version: "encryptionkey@0"
I20260807 11:34:38.872680 13772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:38.892289 13772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:38.896783 13772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:38.897043 13772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:38.898025 13772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:38.899499 13772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:38.899611 13772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:38.899773 13772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:38.899847 13772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:38.936980 13772 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:38161
I20260807 11:34:38.937033 13927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:38161 every 8 connection(s)
I20260807 11:34:38.938445 13772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:34:38.944509 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 13772
I20260807 11:34:38.944840 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:34:38.945490 24989 external_mini_cluster.cc:1428] Setting key 6fb12f91f038a26dab69c341ec2b1206
Aug 07 11:34:38 dist-test-slave-wl2h krb5kdc[13328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102478, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:38.961081 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:38.941237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:36603 (local address 127.24.103.126:41503)
0807 11:34:38.941758 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:38.941762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:38.942534 (+ 772us) server_negotiation.cc:407] Connection header received
0807 11:34:38.944795 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:38.944802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:38.944931 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:38.945044 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:38.947098 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:38.948070 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:38.950098 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:38.950560 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:38.950697 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:38.954567 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:38.954572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:38.954576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:38.954616 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:38.957462 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:38.958332 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:38.958337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:38.958339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:38.958474 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:38.958938 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:38.958942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:38.958944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:38.959342 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:38.959523 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:38.960437 (+ 914us) server_negotiation.cc:299] Negotiation successful
0807 11:34:38.960755 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":298,"threads_started":1}
I20260807 11:34:38.962196 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:38.941643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:41503 (local address 127.24.103.67:36603)
0807 11:34:38.942365 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:38.942403 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:38.944318 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:38.945352 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:38.945365 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:38.946192 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:38.946819 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:38.946840 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:38.948258 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:38.948266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:38.949920 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:38.949934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:38.950815 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:38.950821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:38.951122 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:38.951881 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:34:38.951909 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:38.954310 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:38.957663 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:38.957671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:38.957686 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:38.958145 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:38.958618 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:38.958622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:38.958624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:38.958827 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:38.959492 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:38.959500 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:38.959889 (+ 389us) client_negotiation.cc:769] Sending connection context
0807 11:34:38.960277 (+ 388us) client_negotiation.cc:240] Negotiation successful
0807 11:34:38.960870 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":278,"threads_started":1}
I20260807 11:34:38.963773 13928 heartbeater.cc:347] Connected to a master server at 127.24.103.126:41503
I20260807 11:34:38.964090 13928 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:38.964880 13928 heartbeater.cc:511] Master 127.24.103.126:41503 requested a full tablet report, sending...
I20260807 11:34:38.966637 13381 ts_manager.cc:194] Registered new tserver with Master: 74185555974c4fe6a9b4b360bb62e547 (127.24.103.67:38161)
I20260807 11:34:38.967574 13381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:36603
I20260807 11:34:38.975111 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:34:38.996613 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:38.980909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.24.103.126:41503)
0807 11:34:38.981174 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:38.981186 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:38.981657 (+ 471us) server_negotiation.cc:407] Connection header received
0807 11:34:38.981789 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:38.981793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:38.981872 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:38.981975 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:38.983816 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:38.984581 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:38.986187 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:38.986669 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:38.986788 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:38.989906 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:38.989912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:38.989916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:38.989956 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:38.992963 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:38.993991 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:38.993995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:38.993998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:38.994171 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:38.994749 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:38.994753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:38.994755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:38.995229 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:38.995464 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:38.995899 (+ 435us) server_negotiation.cc:299] Negotiation successful
0807 11:34:38.996066 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 11:34:39.001401 13381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:34:39.009407 13381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:34:39.044169 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.024813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.24.103.67:38161)
0807 11:34:39.025784 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:39.025791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:39.025822 (+ 31us) server_negotiation.cc:407] Connection header received
0807 11:34:39.026038 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:39.026045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:39.026327 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:39.026545 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:39.029267 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.033280 (+ 4013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.039042 (+ 5762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.041161 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.041340 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.043093 (+ 1753us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:39.043568 (+ 475us) server_negotiation.cc:299] Negotiation successful
0807 11:34:39.043776 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":669,"threads_started":1}
I20260807 11:34:39.044297 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.024475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:38161 (local address 127.0.0.1:42246)
0807 11:34:39.025562 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:39.025600 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:39.025795 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:39.026889 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:39.026897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:39.026927 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:39.028982 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:39.029005 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.033500 (+ 4495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.033510 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:39.038731 (+ 5221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:39.038766 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.042971 (+ 4205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.042978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:39.043257 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.043337 (+ 80us) client_negotiation.cc:769] Sending connection context
0807 11:34:39.043502 (+ 165us) client_negotiation.cc:240] Negotiation successful
0807 11:34:39.043756 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":1,"thread_start_us":698,"threads_started":1}
I20260807 11:34:39.048533 13847 tablet_service.cc:1467] Processing CreateTablet for tablet 5563f5879a20455aa351fe8ede443112 (DEFAULT_TABLE table=test-table [id=8bcbf8dba189431e9403dec31e86f142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:39.050228 13847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5563f5879a20455aa351fe8ede443112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:39.051564 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.026034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:34535 (local address 127.0.0.1:45320)
0807 11:34:39.026628 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:39.026649 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:39.026760 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:39.043230 (+ 16470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:39.043235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:39.043264 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:39.043727 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:39.043749 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.046175 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.046181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:39.048512 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:39.048526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.050548 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.050557 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:39.050744 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.050779 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:34:39.050884 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 11:34:39.051073 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":31,"thread_start_us":317,"threads_started":1}
I20260807 11:34:39.051877 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.027273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.24.103.66:37827)
0807 11:34:39.042554 (+ 15281us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:39.042564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:39.043384 (+ 820us) server_negotiation.cc:407] Connection header received
0807 11:34:39.043500 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:39.043509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:39.043726 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:39.044605 (+ 879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:39.044874 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.046460 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.049475 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.050607 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.050753 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.050955 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:39.051321 (+ 366us) server_negotiation.cc:299] Negotiation successful
0807 11:34:39.051509 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15108,"thread_start_us":15293,"threads_started":1}
I20260807 11:34:39.052224 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.026358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.24.103.65:34535)
0807 11:34:39.041728 (+ 15370us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:39.041742 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:39.041795 (+ 53us) server_negotiation.cc:407] Connection header received
0807 11:34:39.041933 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:39.041946 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:39.042319 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:39.043972 (+ 1653us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:39.044264 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.045998 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.048695 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.050292 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.050486 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.050718 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:39.051682 (+ 964us) server_negotiation.cc:299] Negotiation successful
0807 11:34:39.051864 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15102,"thread_start_us":16717,"threads_started":1}
I20260807 11:34:39.052659 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.026742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:37827 (local address 127.0.0.1:51816)
0807 11:34:39.043076 (+ 16334us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:39.043102 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:39.043244 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:39.043946 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:39.043952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:39.043979 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:39.044424 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:39.044436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.046620 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.046628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:39.049253 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:39.049297 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.050892 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.050899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:39.051093 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.051136 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 11:34:39.051240 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 11:34:39.051377 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16139,"mutex_wait_us":21,"thread_start_us":15989,"threads_started":1}
I20260807 11:34:39.056493 13539 tablet_service.cc:1467] Processing CreateTablet for tablet 5563f5879a20455aa351fe8ede443112 (DEFAULT_TABLE table=test-table [id=8bcbf8dba189431e9403dec31e86f142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:39.057307 13691 tablet_service.cc:1467] Processing CreateTablet for tablet 5563f5879a20455aa351fe8ede443112 (DEFAULT_TABLE table=test-table [id=8bcbf8dba189431e9403dec31e86f142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:39.057847 13539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5563f5879a20455aa351fe8ede443112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:39.058634 13691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5563f5879a20455aa351fe8ede443112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:39.065755 13950 tablet_bootstrap.cc:492] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: Bootstrap starting.
I20260807 11:34:39.069631 13950 tablet_bootstrap.cc:654] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:39.071031 13950 log.cc:824] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:39.074662 13953 tablet_bootstrap.cc:492] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: Bootstrap starting.
I20260807 11:34:39.077497 13952 tablet_bootstrap.cc:492] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: Bootstrap starting.
I20260807 11:34:39.078783 13953 tablet_bootstrap.cc:654] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:39.079511 13950 tablet_bootstrap.cc:492] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: No bootstrap required, opened a new log
I20260807 11:34:39.079805 13950 ts_tablet_manager.cc:1397] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260807 11:34:39.080189 13953 log.cc:824] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:39.080785 13952 tablet_bootstrap.cc:654] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:39.082024 13952 log.cc:824] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:39.084728 13952 tablet_bootstrap.cc:492] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: No bootstrap required, opened a new log
I20260807 11:34:39.085019 13952 ts_tablet_manager.cc:1397] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260807 11:34:39.085374 13953 tablet_bootstrap.cc:492] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: No bootstrap required, opened a new log
I20260807 11:34:39.085667 13953 ts_tablet_manager.cc:1397] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260807 11:34:39.090860 13950 raft_consensus.cc:348] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74185555974c4fe6a9b4b360bb62e547" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38161 } } peers { permanent_uuid: "e79860f789304a3ba16bcd4a8a4acffe" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34535 } } peers { permanent_uuid: "84d9a986d6a1426696dca538566c817a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 37827 } }
I20260807 11:34:39.091506 13950 raft_consensus.cc:374] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:39.091615 13950 raft_consensus.cc:729] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74185555974c4fe6a9b4b360bb62e547, State: Initialized, Role: FOLLOWER
I20260807 11:34:39.092342 13950 consensus_queue.cc:260] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74185555974c4fe6a9b4b360bb62e547" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38161 } } peers { permanent_uuid: "e79860f789304a3ba16bcd4a8a4acffe" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34535 } } peers { permanent_uuid: "84d9a986d6a1426696dca538566c817a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 37827 } }
I20260807 11:34:39.092705 13952 raft_consensus.cc:348] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74185555974c4fe6a9b4b360bb62e547" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38161 } } peers { permanent_uuid: "e79860f789304a3ba16bcd4a8a4acffe" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34535 } } peers { permanent_uuid: "84d9a986d6a1426696dca538566c817a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 37827 } }
I20260807 11:34:39.093211 13952 raft_consensus.cc:374] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:39.093334 13952 raft_consensus.cc:729] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79860f789304a3ba16bcd4a8a4acffe, State: Initialized, Role: FOLLOWER
I20260807 11:34:39.093750 13928 heartbeater.cc:503] Master 127.24.103.126:41503 was elected leader, sending a full tablet report...
I20260807 11:34:39.094298 13950 ts_tablet_manager.cc:1428] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260807 11:34:39.095387 13952 consensus_queue.cc:260] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74185555974c4fe6a9b4b360bb62e547" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38161 } } peers { permanent_uuid: "e79860f789304a3ba16bcd4a8a4acffe" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34535 } } peers { permanent_uuid: "84d9a986d6a1426696dca538566c817a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 37827 } }
I20260807 11:34:39.097290 13952 ts_tablet_manager.cc:1428] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260807 11:34:39.096472 13953 raft_consensus.cc:348] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74185555974c4fe6a9b4b360bb62e547" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38161 } } peers { permanent_uuid: "e79860f789304a3ba16bcd4a8a4acffe" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34535 } } peers { permanent_uuid: "84d9a986d6a1426696dca538566c817a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 37827 } }
I20260807 11:34:39.097972 13953 raft_consensus.cc:374] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:39.098097 13953 raft_consensus.cc:729] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d9a986d6a1426696dca538566c817a, State: Initialized, Role: FOLLOWER
I20260807 11:34:39.098757 13953 consensus_queue.cc:260] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74185555974c4fe6a9b4b360bb62e547" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38161 } } peers { permanent_uuid: "e79860f789304a3ba16bcd4a8a4acffe" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34535 } } peers { permanent_uuid: "84d9a986d6a1426696dca538566c817a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 37827 } }
I20260807 11:34:39.100216 13766 heartbeater.cc:503] Master 127.24.103.126:41503 was elected leader, sending a full tablet report...
I20260807 11:34:39.100845 13953 ts_tablet_manager.cc:1428] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
W20260807 11:34:39.127053 13767 tablet.cc:2369] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:34:39.190323 13929 tablet.cc:2369] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:34:39.226459 13957 raft_consensus.cc:482] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:34:39.226837 13957 raft_consensus.cc:504] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74185555974c4fe6a9b4b360bb62e547" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38161 } } peers { permanent_uuid: "e79860f789304a3ba16bcd4a8a4acffe" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34535 } } peers { permanent_uuid: "84d9a986d6a1426696dca538566c817a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 37827 } }
I20260807 11:34:39.229020 13957 leader_election.cc:288] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74185555974c4fe6a9b4b360bb62e547 (127.24.103.67:38161), 84d9a986d6a1426696dca538566c817a (127.24.103.66:37827)
I20260807 11:34:39.238739 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.229610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:49003 (local address 127.24.103.67:38161)
0807 11:34:39.229903 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:39.229912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:39.230476 (+ 564us) server_negotiation.cc:407] Connection header received
0807 11:34:39.230725 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:39.230732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:39.230826 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:39.230897 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:39.231778 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.233005 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.236238 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.237686 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.237884 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.237968 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:39.238293 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 11:34:39.238441 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 11:34:39.238873 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.229516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:38161 (local address 127.24.103.65:49003)
0807 11:34:39.230311 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:39.230343 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:39.230513 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:39.231138 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:39.231143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:39.231174 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:39.231609 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:39.231626 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.233239 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.233245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:39.236024 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:39.236042 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.237915 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.237920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:39.238113 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.238143 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 11:34:39.238234 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 11:34:39.238383 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":168,"thread_start_us":406,"threads_started":1}
I20260807 11:34:39.239089 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.230712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:55705 (local address 127.24.103.66:37827)
0807 11:34:39.230910 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:39.230914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:39.231145 (+ 231us) server_negotiation.cc:407] Connection header received
0807 11:34:39.231243 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:39.231246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:39.231311 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:39.231387 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:39.232042 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.233078 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.236229 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.237730 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.237884 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.237968 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:39.238568 (+ 600us) server_negotiation.cc:299] Negotiation successful
0807 11:34:39.238707 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:34:39.239686 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.230345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:37827 (local address 127.24.103.65:55705)
0807 11:34:39.230976 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:39.230993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:39.231138 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:39.231431 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:39.231438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:39.231471 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:39.231871 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:39.231880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.233203 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.233209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:39.236025 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:39.236042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.238155 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.238161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:39.238381 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.238417 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 11:34:39.238524 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 11:34:39.238672 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":162,"thread_start_us":305,"threads_started":1}
I20260807 11:34:39.240091 13873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5563f5879a20455aa351fe8ede443112" candidate_uuid: "e79860f789304a3ba16bcd4a8a4acffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74185555974c4fe6a9b4b360bb62e547" is_pre_election: true
I20260807 11:34:39.240724 13873 raft_consensus.cc:2454] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79860f789304a3ba16bcd4a8a4acffe in term 0.
I20260807 11:34:39.240758 13715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5563f5879a20455aa351fe8ede443112" candidate_uuid: "e79860f789304a3ba16bcd4a8a4acffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d9a986d6a1426696dca538566c817a" is_pre_election: true
I20260807 11:34:39.241302 13715 raft_consensus.cc:2454] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79860f789304a3ba16bcd4a8a4acffe in term 0.
I20260807 11:34:39.241582 13480 leader_election.cc:302] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74185555974c4fe6a9b4b360bb62e547, e79860f789304a3ba16bcd4a8a4acffe; no voters:
I20260807 11:34:39.242074 13957 raft_consensus.cc:2781] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:34:39.242205 13957 raft_consensus.cc:482] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:34:39.242350 13957 raft_consensus.cc:3037] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:39.244765 13957 raft_consensus.cc:504] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74185555974c4fe6a9b4b360bb62e547" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38161 } } peers { permanent_uuid: "e79860f789304a3ba16bcd4a8a4acffe" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34535 } } peers { permanent_uuid: "84d9a986d6a1426696dca538566c817a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 37827 } }
I20260807 11:34:39.245394 13957 leader_election.cc:288] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [CANDIDATE]: Term 1 election: Requested vote from peers 74185555974c4fe6a9b4b360bb62e547 (127.24.103.67:38161), 84d9a986d6a1426696dca538566c817a (127.24.103.66:37827)
I20260807 11:34:39.245864 13873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5563f5879a20455aa351fe8ede443112" candidate_uuid: "e79860f789304a3ba16bcd4a8a4acffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74185555974c4fe6a9b4b360bb62e547"
I20260807 11:34:39.246062 13873 raft_consensus.cc:3037] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:39.246024 13715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5563f5879a20455aa351fe8ede443112" candidate_uuid: "e79860f789304a3ba16bcd4a8a4acffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d9a986d6a1426696dca538566c817a"
I20260807 11:34:39.246279 13715 raft_consensus.cc:3037] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:39.248638 13873 raft_consensus.cc:2454] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79860f789304a3ba16bcd4a8a4acffe in term 1.
I20260807 11:34:39.248911 13715 raft_consensus.cc:2454] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79860f789304a3ba16bcd4a8a4acffe in term 1.
I20260807 11:34:39.249204 13480 leader_election.cc:302] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74185555974c4fe6a9b4b360bb62e547, e79860f789304a3ba16bcd4a8a4acffe; no voters:
I20260807 11:34:39.249583 13957 raft_consensus.cc:2781] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:39.250082 13957 raft_consensus.cc:686] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 1 LEADER]: Becoming Leader. State: Replica: e79860f789304a3ba16bcd4a8a4acffe, State: Running, Role: LEADER
I20260807 11:34:39.250681 13957 consensus_queue.cc:237] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74185555974c4fe6a9b4b360bb62e547" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38161 } } peers { permanent_uuid: "e79860f789304a3ba16bcd4a8a4acffe" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34535 } } peers { permanent_uuid: "84d9a986d6a1426696dca538566c817a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 37827 } }
I20260807 11:34:39.256039 13379 catalog_manager.cc:5495] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe reported cstate change: term changed from 0 to 1, leader changed from <none> to e79860f789304a3ba16bcd4a8a4acffe (127.24.103.65). New cstate: current_term: 1 leader_uuid: "e79860f789304a3ba16bcd4a8a4acffe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74185555974c4fe6a9b4b360bb62e547" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79860f789304a3ba16bcd4a8a4acffe" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d9a986d6a1426696dca538566c817a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 37827 } health_report { overall_health: UNKNOWN } } }
I20260807 11:34:39.289176 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.283258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.24.103.65:34535)
0807 11:34:39.283524 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:39.283529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:39.283546 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:34:39.283632 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:39.283636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:39.283745 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:39.283930 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:34:39.284425 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.285195 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.286688 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.287216 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.287325 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.287907 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:34:39.288456 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:34:39.288615 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:39.288741 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 11:34:39.288831 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:34:39.298606 13715 raft_consensus.cc:1261] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a [term 1 FOLLOWER]: Refusing update from remote peer e79860f789304a3ba16bcd4a8a4acffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:39.298605 13873 raft_consensus.cc:1261] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547 [term 1 FOLLOWER]: Refusing update from remote peer e79860f789304a3ba16bcd4a8a4acffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:39.299726 13957 consensus_queue.cc:1035] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d9a986d6a1426696dca538566c817a" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 37827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:34:39.300099 13963 consensus_queue.cc:1035] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "74185555974c4fe6a9b4b360bb62e547" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 11:34:39.307104 13612 tablet.cc:2369] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:34:39.313679 13966 mvcc.cc:205] Tried to move back new op lower bound from 7315875755194720256 to 7315875755014963200. Current Snapshot: MvccSnapshot[applied={T|T < 7315875755194720256}]
I20260807 11:34:39.331414 13379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:39.331734 13379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:39.335888 13379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ba3e83c71be45039b3e9ef17dc1df28: Sending DeleteTablet for 3 replicas of tablet 5563f5879a20455aa351fe8ede443112
I20260807 11:34:39.337174 13847 tablet_service.cc:1514] Processing DeleteTablet for tablet 5563f5879a20455aa351fe8ede443112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:39 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:42246
I20260807 11:34:39.337390 13691 tablet_service.cc:1514] Processing DeleteTablet for tablet 5563f5879a20455aa351fe8ede443112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:39 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:51816
I20260807 11:34:39.337389 13539 tablet_service.cc:1514] Processing DeleteTablet for tablet 5563f5879a20455aa351fe8ede443112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:39 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:45320
I20260807 11:34:39.339236 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 13462
I20260807 11:34:39.358157 13978 tablet_replica.cc:330] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: stopping tablet replica
I20260807 11:34:39.358822 13978 raft_consensus.cc:2229] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:39.359557 13978 raft_consensus.cc:2258] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:39.360491 13977 tablet_replica.cc:330] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: stopping tablet replica
I20260807 11:34:39.360948 13976 tablet_replica.cc:330] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: stopping tablet replica
I20260807 11:34:39.360958 13977 raft_consensus.cc:2229] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:34:39.361356 13976 raft_consensus.cc:2229] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:39.361683 13977 raft_consensus.cc:2258] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:39.361871 13976 raft_consensus.cc:2258] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:39.363934 13978 ts_tablet_manager.cc:1905] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:39.364318 13977 ts_tablet_manager.cc:1905] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:39.364605 13976 ts_tablet_manager.cc:1905] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 11:34:39 dist-test-slave-wl2h krb5kdc[13328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:34:39.371486 13978 ts_tablet_manager.cc:1918] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:39.371727 13978 log.cc:1196] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/5563f5879a20455aa351fe8ede443112
I20260807 11:34:39.372081 13977 ts_tablet_manager.cc:1918] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:39.372272 13977 log.cc:1196] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/5563f5879a20455aa351fe8ede443112
I20260807 11:34:39.372346 13978 ts_tablet_manager.cc:1939] T 5563f5879a20455aa351fe8ede443112 P 84d9a986d6a1426696dca538566c817a: Deleting consensus metadata
I20260807 11:34:39.372928 13977 ts_tablet_manager.cc:1939] T 5563f5879a20455aa351fe8ede443112 P e79860f789304a3ba16bcd4a8a4acffe: Deleting consensus metadata
I20260807 11:34:39.374429 13363 catalog_manager.cc:4841] TS 84d9a986d6a1426696dca538566c817a (127.24.103.66:37827): tablet 5563f5879a20455aa351fe8ede443112 (table test-table [id=8bcbf8dba189431e9403dec31e86f142]) successfully deleted
I20260807 11:34:39.374804 13363 catalog_manager.cc:4841] TS e79860f789304a3ba16bcd4a8a4acffe (127.24.103.65:34535): tablet 5563f5879a20455aa351fe8ede443112 (table test-table [id=8bcbf8dba189431e9403dec31e86f142]) successfully deleted
I20260807 11:34:39.384109 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.340400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.24.103.65:34535)
0807 11:34:39.340606 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:39.340612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:39.347538 (+ 6926us) server_negotiation.cc:407] Connection header received
0807 11:34:39.347764 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:39.347770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:39.347866 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:39.347976 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:39.355559 (+ 7583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.357009 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.360472 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.361318 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.361463 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.375243 (+ 13780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:39.375261 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:39.375278 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:39.375340 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:39.380075 (+ 4735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:39.381157 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:39.381167 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:39.381172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:39.381350 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:39.381851 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:39.381857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:39.381861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:39.382319 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:39.382635 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:39.383321 (+ 686us) server_negotiation.cc:299] Negotiation successful
0807 11:34:39.383619 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 11:34:39.385694 13606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:39.389258 13976 ts_tablet_manager.cc:1918] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:39.389503 13976 log.cc:1196] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/5563f5879a20455aa351fe8ede443112
I20260807 11:34:39.390384 13976 ts_tablet_manager.cc:1939] T 5563f5879a20455aa351fe8ede443112 P 74185555974c4fe6a9b4b360bb62e547: Deleting consensus metadata
I20260807 11:34:39.392707 13363 catalog_manager.cc:4841] TS 74185555974c4fe6a9b4b360bb62e547 (127.24.103.67:38161): tablet 5563f5879a20455aa351fe8ede443112 (table test-table [id=8bcbf8dba189431e9403dec31e86f142]) successfully deleted
I20260807 11:34:39.488219 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 13462
I20260807 11:34:39.513089 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 13617
Aug 07 11:34:39 dist-test-slave-wl2h krb5kdc[13328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:34:39.529466 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.513982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51824 (local address 127.24.103.66:37827)
0807 11:34:39.514200 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:39.514205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:39.514226 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:34:39.514432 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:39.514437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:39.514513 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:39.514625 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:39.516304 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.517084 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.518495 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.518883 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.518981 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.523530 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:39.523545 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:39.523562 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:39.523597 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:39.526369 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:39.527248 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:39.527260 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:39.527266 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:39.527413 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:39.527790 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:39.527794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:39.527796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:39.528256 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:39.528492 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:39.528939 (+ 447us) server_negotiation.cc:299] Negotiation successful
0807 11:34:39.529147 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 11:34:39.530197 13759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:39.624984 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 13617
I20260807 11:34:39.646507 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 13772
Aug 07 11:34:39 dist-test-slave-wl2h krb5kdc[13328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:34:39.664052 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:39.647464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.24.103.67:38161)
0807 11:34:39.647693 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:39.647698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:39.647719 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:34:39.647787 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:39.647791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:39.647876 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:39.647977 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:39.649444 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.650179 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.651826 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:39.652305 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:39.652416 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:39.657675 (+ 5259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:39.657693 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:39.657711 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:39.657761 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:39.661048 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:39.661816 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:39.661824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:39.661827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:39.661992 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:39.662373 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:39.662376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:39.662378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:39.662760 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:39.662992 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:39.663556 (+ 564us) server_negotiation.cc:299] Negotiation successful
0807 11:34:39.663745 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 11:34:39.664927 13923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:39.757371 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 13772
I20260807 11:34:39.778294 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 13344
I20260807 11:34:39.779542 13422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:39.894582 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 13344
2026-08-07T11:34:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4832 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T11:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:34:39Z Disabled control of system clock
I20260807 11:34:39.996189 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:45425
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:45425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:34:40.351368 13989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:40.351871 13989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:40.352144 13989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:40.360785 13989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:34:40.360913 13989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:40.360953 13989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:40.361061 13989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:34:40.361140 13989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:34:40.371652 13989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:45425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:45425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:40.373059 13989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:40.374902 13989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:40.387380 13997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:40.387416 13996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:40.387506 13999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:40.388057 13989 server_base.cc:1034] running on GCE node
I20260807 11:34:40.389000 13989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:40.390564 13989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:40.392041 13989 hybrid_clock.cc:648] HybridClock initialized: now 1786102480391684 us; error 297 us; skew 500 ppm
I20260807 11:34:40.392753 13989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:40.395519 13989 webserver.cc:460] Webserver started at http://127.24.103.126:38633/ using document root <none> and password file <none>
I20260807 11:34:40.396332 13989 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:40.396445 13989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:40.396828 13989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:40.400242 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "a73dd9728ccd464daa17767cb9621102"
format_stamp: "Formatted at 2026-08-07 11:34:40 on dist-test-slave-wl2h"
server_key: "c7dca9ab14a657936bb6e215ca8fc3d4"
server_key_iv: "a24b74ca93c58ae573970a3eb3fcd668"
server_key_version: "encryptionkey@0"
I20260807 11:34:40.401206 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "a73dd9728ccd464daa17767cb9621102"
format_stamp: "Formatted at 2026-08-07 11:34:40 on dist-test-slave-wl2h"
server_key: "c7dca9ab14a657936bb6e215ca8fc3d4"
server_key_iv: "a24b74ca93c58ae573970a3eb3fcd668"
server_key_version: "encryptionkey@0"
I20260807 11:34:40.407181 13989 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 11:34:40.411381 14005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:40.412885 13989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260807 11:34:40.413090 13989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "a73dd9728ccd464daa17767cb9621102"
format_stamp: "Formatted at 2026-08-07 11:34:40 on dist-test-slave-wl2h"
server_key: "c7dca9ab14a657936bb6e215ca8fc3d4"
server_key_iv: "a24b74ca93c58ae573970a3eb3fcd668"
server_key_version: "encryptionkey@0"
I20260807 11:34:40.413342 13989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:40.436326 13989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:40.437228 13989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:40.437533 13989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:40.458096 13989 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:45425
I20260807 11:34:40.458141 14062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:45425 every 8 connection(s)
I20260807 11:34:40.459540 13989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:34:40.460646 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 13989
I20260807 11:34:40.460829 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:34:40.461262 24989 external_mini_cluster.cc:1428] Setting key edf683813e8c7db9419cc83fe0a5e9fe
I20260807 11:34:40.464507 14063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:40.478820 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:40.463370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.24.103.126:45425)
0807 11:34:40.464535 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:40.464554 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:40.464614 (+ 60us) server_negotiation.cc:407] Connection header received
0807 11:34:40.467292 (+ 2678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:40.467324 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:40.467959 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:40.468493 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:40.469073 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:40.471471 (+ 2398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:40.472964 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:40.474107 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:40.474320 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:40.475614 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:40.475626 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:40.475646 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:40.476480 (+ 834us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:40.476586 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:40.476628 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:40.476763 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:40.477007 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 11:34:40.477406 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":374,"threads_started":1}
I20260807 11:34:40.480360 14063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102: Bootstrap starting.
I20260807 11:34:40.483932 14063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:40.485064 14063 log.cc:824] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:40.488515 14063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102: No bootstrap required, opened a new log
I20260807 11:34:40.496344 14063 raft_consensus.cc:348] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73dd9728ccd464daa17767cb9621102" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45425 } }
I20260807 11:34:40.496743 14063 raft_consensus.cc:374] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:40.496865 14063 raft_consensus.cc:729] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a73dd9728ccd464daa17767cb9621102, State: Initialized, Role: FOLLOWER
I20260807 11:34:40.497537 14063 consensus_queue.cc:260] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73dd9728ccd464daa17767cb9621102" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45425 } }
I20260807 11:34:40.497782 14063 raft_consensus.cc:388] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:34:40.497944 14063 raft_consensus.cc:482] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:34:40.498140 14063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:40.500522 14063 raft_consensus.cc:504] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73dd9728ccd464daa17767cb9621102" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45425 } }
I20260807 11:34:40.501031 14063 leader_election.cc:302] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a73dd9728ccd464daa17767cb9621102; no voters:
I20260807 11:34:40.501600 14063 leader_election.cc:288] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:34:40.501811 14070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:40.503219 14070 raft_consensus.cc:686] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [term 1 LEADER]: Becoming Leader. State: Replica: a73dd9728ccd464daa17767cb9621102, State: Running, Role: LEADER
I20260807 11:34:40.503894 14063 sys_catalog.cc:564] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:34:40.503818 14070 consensus_queue.cc:237] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73dd9728ccd464daa17767cb9621102" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45425 } }
I20260807 11:34:40.506929 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a73dd9728ccd464daa17767cb9621102. Latest consensus state: current_term: 1 leader_uuid: "a73dd9728ccd464daa17767cb9621102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73dd9728ccd464daa17767cb9621102" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45425 } } }
I20260807 11:34:40.507102 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a73dd9728ccd464daa17767cb9621102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73dd9728ccd464daa17767cb9621102" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45425 } } }
I20260807 11:34:40.507416 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:40.507444 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:40.510321 14080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:34:40.514729 14080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:34:40.523639 14080 catalog_manager.cc:1347] Generated new cluster ID: 83f3abbc02e4405ab4a4e5da377cc02c
I20260807 11:34:40.523788 14080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:34:40.531492 14080 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:34:40.533288 14080 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:34:40.542606 14080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102: Generated new TSK 0
I20260807 11:34:40.543753 14080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:34:40.561777 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45425
--builtin_ntp_servers=127.24.103.84:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:34:40.919878 14090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:40.920198 14090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:40.920411 14090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:40.927654 14090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:40.927795 14090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:40.927997 14090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:34:40.937107 14090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:40.938684 14090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:40.940601 14090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:40.954285 14097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:40.954227 14098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:40.954162 14100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:40.955153 14090 server_base.cc:1034] running on GCE node
I20260807 11:34:40.955806 14090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:40.956792 14090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:40.958003 14090 hybrid_clock.cc:648] HybridClock initialized: now 1786102480957945 us; error 53 us; skew 500 ppm
I20260807 11:34:40.958343 14090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:40.960906 14090 webserver.cc:460] Webserver started at http://127.24.103.65:34207/ using document root <none> and password file <none>
I20260807 11:34:40.961670 14090 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:40.961766 14090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:40.962245 14090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:40.965638 14090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "c2de2dd9a6f5464594875b9e2af32aaa"
format_stamp: "Formatted at 2026-08-07 11:34:40 on dist-test-slave-wl2h"
server_key: "62b2323980e30a8d240c3c55f156d013"
server_key_iv: "05da15e33e0ec954a87c422b30eb3a96"
server_key_version: "encryptionkey@0"
I20260807 11:34:40.966413 14090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "c2de2dd9a6f5464594875b9e2af32aaa"
format_stamp: "Formatted at 2026-08-07 11:34:40 on dist-test-slave-wl2h"
server_key: "62b2323980e30a8d240c3c55f156d013"
server_key_iv: "05da15e33e0ec954a87c422b30eb3a96"
server_key_version: "encryptionkey@0"
I20260807 11:34:40.971752 14090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 11:34:40.975389 14106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:40.976928 14090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:34:40.977070 14090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "c2de2dd9a6f5464594875b9e2af32aaa"
format_stamp: "Formatted at 2026-08-07 11:34:40 on dist-test-slave-wl2h"
server_key: "62b2323980e30a8d240c3c55f156d013"
server_key_iv: "05da15e33e0ec954a87c422b30eb3a96"
server_key_version: "encryptionkey@0"
I20260807 11:34:40.977291 14090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:40.994184 14090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:40.995167 14090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:40.995484 14090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:40.996541 14090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:40.998029 14090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:40.998119 14090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:40.998198 14090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:40.998262 14090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:41.034937 14090 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:37585
I20260807 11:34:41.034993 14241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:37585 every 8 connection(s)
I20260807 11:34:41.036352 14090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:34:41.036849 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 14090
I20260807 11:34:41.037060 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:34:41.037703 24989 external_mini_cluster.cc:1428] Setting key 48981813aac920a70e26167fdb7cfa39
I20260807 11:34:41.041164 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45425
--builtin_ntp_servers=127.24.103.84:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:34:41.050837 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:41.039246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:43537 (local address 127.24.103.126:45425)
0807 11:34:41.039731 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:41.039738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:41.041157 (+ 1419us) server_negotiation.cc:407] Connection header received
0807 11:34:41.043163 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:41.043168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:41.043270 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:41.043388 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:41.044443 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:41.045140 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:41.047698 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:41.048205 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:41.048317 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:41.050035 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:41.050039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:41.050043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:41.050063 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:41.050105 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:41.050118 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:41.050191 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:41.050464 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 11:34:41.050553 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":229,"threads_started":1}
I20260807 11:34:41.052368 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:41.040022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45425 (local address 127.24.103.65:43537)
0807 11:34:41.040960 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:41.041007 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:41.042865 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:41.043625 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:41.043633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:41.043716 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:41.044274 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:41.044290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:41.045356 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:41.045363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:41.047511 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:41.047530 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:41.048450 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:41.048459 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:41.048812 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:41.049652 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:34:41.049658 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:41.049713 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:34:41.049817 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:41.050221 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:41.050235 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:41.050246 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 11:34:41.050437 (+ 191us) client_negotiation.cc:240] Negotiation successful
0807 11:34:41.050880 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":314,"threads_started":1}
I20260807 11:34:41.053968 14242 heartbeater.cc:347] Connected to a master server at 127.24.103.126:45425
I20260807 11:34:41.054291 14242 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:41.055190 14242 heartbeater.cc:511] Master 127.24.103.126:45425 requested a full tablet report, sending...
I20260807 11:34:41.057468 14026 ts_manager.cc:194] Registered new tserver with Master: c2de2dd9a6f5464594875b9e2af32aaa (127.24.103.65:37585)
I20260807 11:34:41.058949 14026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.65:43537
W20260807 11:34:41.408197 14246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:41.408552 14246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:41.408726 14246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:41.416642 14246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:41.416793 14246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:41.416981 14246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:34:41.426590 14246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:41.428050 14246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:41.429998 14246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:41.444693 14253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:41.444947 14254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:41.444973 14256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:41.446538 14246 server_base.cc:1034] running on GCE node
I20260807 11:34:41.447293 14246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:41.448348 14246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:41.449580 14246 hybrid_clock.cc:648] HybridClock initialized: now 1786102481449532 us; error 36 us; skew 500 ppm
I20260807 11:34:41.449959 14246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:41.452605 14246 webserver.cc:460] Webserver started at http://127.24.103.66:38583/ using document root <none> and password file <none>
I20260807 11:34:41.453346 14246 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:41.453441 14246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:41.453819 14246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:41.457010 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "a5acb451bd844e7e99c31163119bab6b"
format_stamp: "Formatted at 2026-08-07 11:34:41 on dist-test-slave-wl2h"
server_key: "fafc1aa5d083b2e9532aedfec9ace796"
server_key_iv: "a1b62696ce604837e2195bc54e4a3da2"
server_key_version: "encryptionkey@0"
I20260807 11:34:41.458020 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "a5acb451bd844e7e99c31163119bab6b"
format_stamp: "Formatted at 2026-08-07 11:34:41 on dist-test-slave-wl2h"
server_key: "fafc1aa5d083b2e9532aedfec9ace796"
server_key_iv: "a1b62696ce604837e2195bc54e4a3da2"
server_key_version: "encryptionkey@0"
I20260807 11:34:41.463742 14246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 11:34:41.467700 14262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:41.469482 14246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:34:41.469668 14246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "a5acb451bd844e7e99c31163119bab6b"
format_stamp: "Formatted at 2026-08-07 11:34:41 on dist-test-slave-wl2h"
server_key: "fafc1aa5d083b2e9532aedfec9ace796"
server_key_iv: "a1b62696ce604837e2195bc54e4a3da2"
server_key_version: "encryptionkey@0"
I20260807 11:34:41.469911 14246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:41.491133 14246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:41.492090 14246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:41.492431 14246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:41.493510 14246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:41.495141 14246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:41.495253 14246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:41.495390 14246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:41.495484 14246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:41.530557 14246 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:41431
I20260807 11:34:41.530658 14393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:41431 every 8 connection(s)
I20260807 11:34:41.531935 14246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:34:41.538626 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 14246
I20260807 11:34:41.538882 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:34:41.539516 24989 external_mini_cluster.cc:1428] Setting key d0d6308ffaa998c37900c7d4e386cdbc
I20260807 11:34:41.542779 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45425
--builtin_ntp_servers=127.24.103.84:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:34:41.546047 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:41.534694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:54171 (local address 127.24.103.126:45425)
0807 11:34:41.534921 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:41.534927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:41.536458 (+ 1531us) server_negotiation.cc:407] Connection header received
0807 11:34:41.538494 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:41.538503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:41.538589 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:41.538688 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:41.540228 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:41.541091 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:41.542990 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:41.543509 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:41.543587 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:41.545205 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:41.545213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:41.545222 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:41.545242 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:41.545275 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:41.545287 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:41.545393 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:41.545622 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 11:34:41.545736 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:34:41.547197 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:41.535288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45425 (local address 127.24.103.66:54171)
0807 11:34:41.536284 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:41.536329 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:41.537955 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:41.539029 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:41.539043 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:41.539166 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:41.540000 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:41.540027 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:41.541297 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:41.541301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:41.542750 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:41.542763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:41.543833 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:41.543837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:41.544033 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:41.544828 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:34:41.544831 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:41.544871 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:34:41.544960 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:41.545393 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:41.545407 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:41.545418 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 11:34:41.545611 (+ 193us) client_negotiation.cc:240] Negotiation successful
0807 11:34:41.546029 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":340,"threads_started":1}
I20260807 11:34:41.548859 14394 heartbeater.cc:347] Connected to a master server at 127.24.103.126:45425
I20260807 11:34:41.549220 14394 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:41.550242 14394 heartbeater.cc:511] Master 127.24.103.126:45425 requested a full tablet report, sending...
I20260807 11:34:41.551983 14026 ts_manager.cc:194] Registered new tserver with Master: a5acb451bd844e7e99c31163119bab6b (127.24.103.66:41431)
I20260807 11:34:41.552883 14026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.66:54171
W20260807 11:34:41.905831 14398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:41.906150 14398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:41.906366 14398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:41.914113 14398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:41.914237 14398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:41.914412 14398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:34:41.924121 14398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:45425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:41.925592 14398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:41.927582 14398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:41.941823 14403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:41.941660 14406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:41.941746 14404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:41.942947 14398 server_base.cc:1034] running on GCE node
I20260807 11:34:41.943647 14398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:41.944698 14398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:41.945932 14398 hybrid_clock.cc:648] HybridClock initialized: now 1786102481945887 us; error 28 us; skew 500 ppm
I20260807 11:34:41.946287 14398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:41.948812 14398 webserver.cc:460] Webserver started at http://127.24.103.67:44481/ using document root <none> and password file <none>
I20260807 11:34:41.949539 14398 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:41.949667 14398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:41.950016 14398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:41.953181 14398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "e992cf00a7064ae8b56af666f535120a"
format_stamp: "Formatted at 2026-08-07 11:34:41 on dist-test-slave-wl2h"
server_key: "e88ed58f3f96e3d897f54c8a3c9f297f"
server_key_iv: "f4c3cc882a80b82d26f451b806fd0eed"
server_key_version: "encryptionkey@0"
I20260807 11:34:41.954051 14398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "e992cf00a7064ae8b56af666f535120a"
format_stamp: "Formatted at 2026-08-07 11:34:41 on dist-test-slave-wl2h"
server_key: "e88ed58f3f96e3d897f54c8a3c9f297f"
server_key_iv: "f4c3cc882a80b82d26f451b806fd0eed"
server_key_version: "encryptionkey@0"
I20260807 11:34:41.959720 14398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 11:34:41.963704 14413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:41.965271 14398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:34:41.965492 14398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "e992cf00a7064ae8b56af666f535120a"
format_stamp: "Formatted at 2026-08-07 11:34:41 on dist-test-slave-wl2h"
server_key: "e88ed58f3f96e3d897f54c8a3c9f297f"
server_key_iv: "f4c3cc882a80b82d26f451b806fd0eed"
server_key_version: "encryptionkey@0"
I20260807 11:34:41.965727 14398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:41.998979 14398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:41.999972 14398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:42.000313 14398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:42.001420 14398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:42.003064 14398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:42.003158 14398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:42.003278 14398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:42.003345 14398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:42.039645 14398 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:36331
I20260807 11:34:42.039711 14545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:36331 every 8 connection(s)
I20260807 11:34:42.041016 14398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:34:42.050477 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 14398
I20260807 11:34:42.050675 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:34:42.051282 24989 external_mini_cluster.cc:1428] Setting key c2a4ffa515bcc9f2bddf66a016b50355
I20260807 11:34:42.054970 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.043802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:32845 (local address 127.24.103.126:45425)
0807 11:34:42.044026 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:42.044032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:42.045441 (+ 1409us) server_negotiation.cc:407] Connection header received
0807 11:34:42.047307 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:42.047312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:42.047384 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:42.047494 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:42.048576 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.049359 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.051690 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.052295 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.052456 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.054107 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:42.054113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:42.054118 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:42.054141 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:42.054179 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:42.054196 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:42.054310 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:42.054616 (+ 306us) server_negotiation.cc:299] Negotiation successful
0807 11:34:42.054695 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:34:42.056100 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.044391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:45425 (local address 127.24.103.67:32845)
0807 11:34:42.045283 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.045318 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:42.046941 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:42.047691 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:42.047699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:42.047781 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:42.048395 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.048412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.049555 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.049559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.051532 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.051549 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.052561 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.052568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.052930 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.053788 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:34:42.053791 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:42.053833 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:34:42.053915 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:42.054347 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:42.054364 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:42.054370 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 11:34:42.054604 (+ 234us) client_negotiation.cc:240] Negotiation successful
0807 11:34:42.054933 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":281,"threads_started":1}
I20260807 11:34:42.057494 14546 heartbeater.cc:347] Connected to a master server at 127.24.103.126:45425
I20260807 11:34:42.057849 14546 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:42.058676 14546 heartbeater.cc:511] Master 127.24.103.126:45425 requested a full tablet report, sending...
I20260807 11:34:42.060163 14026 ts_manager.cc:194] Registered new tserver with Master: e992cf00a7064ae8b56af666f535120a (127.24.103.67:36331)
I20260807 11:34:42.060961 14026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.67:32845
I20260807 11:34:42.062978 14242 heartbeater.cc:503] Master 127.24.103.126:45425 was elected leader, sending a full tablet report...
I20260807 11:34:42.067119 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:34:42.077904 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.071605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.24.103.126:45425)
0807 11:34:42.071814 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:42.071818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:42.072233 (+ 415us) server_negotiation.cc:407] Connection header received
0807 11:34:42.072371 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:42.072375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:42.072437 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:42.072525 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:42.073017 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.073818 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.075444 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.075869 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.075973 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.077112 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:42.077116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:42.077119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:42.077137 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:42.077171 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:42.077185 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:42.077304 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:42.077475 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 11:34:42.077560 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 11:34:42.081987 14026 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:34:42.088857 14026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:34:42.115906 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.102429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.24.103.67:36331)
0807 11:34:42.107390 (+ 4961us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:42.107400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:42.107440 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:34:42.107712 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:42.107719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:42.107910 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:42.108085 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:42.108793 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.110276 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.112958 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.113997 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.114148 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.114427 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:42.115324 (+ 897us) server_negotiation.cc:299] Negotiation successful
0807 11:34:42.115491 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4721,"thread_start_us":4013,"threads_started":1}
I20260807 11:34:42.115887 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.101951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:36331 (local address 127.0.0.1:37158)
0807 11:34:42.107311 (+ 5360us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.107336 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:42.107476 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:42.108213 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:42.108218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:42.108242 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:42.108564 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.108574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.110459 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.110464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.112783 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.112794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.114670 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.114678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.115090 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.115145 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 11:34:42.115281 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 11:34:42.115475 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5155,"mutex_wait_us":282,"thread_start_us":5624,"threads_started":1}
I20260807 11:34:42.115949 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.103244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34888 (local address 127.24.103.65:37585)
0807 11:34:42.105359 (+ 2115us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:42.105370 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:42.105408 (+ 38us) server_negotiation.cc:407] Connection header received
0807 11:34:42.105519 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:42.105530 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:42.105811 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:42.106028 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:42.107597 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.109336 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.112414 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.113998 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.114139 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.114430 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:42.115412 (+ 982us) server_negotiation.cc:299] Negotiation successful
0807 11:34:42.115568 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1921,"thread_start_us":1771,"threads_started":1}
I20260807 11:34:42.115934 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.102974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.24.103.66:41431)
0807 11:34:42.103975 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:42.103989 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:42.108032 (+ 4043us) server_negotiation.cc:407] Connection header received
0807 11:34:42.108192 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:42.108201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:42.108407 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:42.108597 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:42.109132 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.110524 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.112672 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.114006 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.114139 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.114400 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:42.115381 (+ 981us) server_negotiation.cc:299] Negotiation successful
0807 11:34:42.115555 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":612,"threads_started":1}
I20260807 11:34:42.116643 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.102428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:41431 (local address 127.0.0.1:49052)
0807 11:34:42.107882 (+ 5454us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.107900 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:42.108029 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:42.108591 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:42.108595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:42.108623 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:42.108966 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.108976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.110685 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.110690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.112414 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.112429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.114552 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.114561 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.114981 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.115165 (+ 184us) client_negotiation.cc:769] Sending connection context
0807 11:34:42.115272 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 11:34:42.116267 (+ 995us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5341,"mutex_wait_us":202,"thread_start_us":1773,"threads_started":1}
I20260807 11:34:42.117177 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.101907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37585 (local address 127.0.0.1:34888)
0807 11:34:42.104702 (+ 2795us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.104765 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:42.105000 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:42.106651 (+ 1651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:42.106661 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:42.106707 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:42.107329 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.107349 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.109510 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.109515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.112184 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.112200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.114630 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.114641 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.115068 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.115165 (+ 97us) client_negotiation.cc:769] Sending connection context
0807 11:34:42.115256 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 11:34:42.116896 (+ 1640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2446,"mutex_wait_us":1496,"thread_start_us":4938,"threads_started":1}
I20260807 11:34:42.120522 14468 tablet_service.cc:1467] Processing CreateTablet for tablet 0866f0e8c2024fcaabe3cf5a1c664725 (DEFAULT_TABLE table=test-table [id=0a46846f2aa44e5c8e2c30ac25ac3048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:42.120524 14167 tablet_service.cc:1467] Processing CreateTablet for tablet 0866f0e8c2024fcaabe3cf5a1c664725 (DEFAULT_TABLE table=test-table [id=0a46846f2aa44e5c8e2c30ac25ac3048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:42.121820 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0866f0e8c2024fcaabe3cf5a1c664725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:42.121829 14167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0866f0e8c2024fcaabe3cf5a1c664725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:42.128432 14318 tablet_service.cc:1467] Processing CreateTablet for tablet 0866f0e8c2024fcaabe3cf5a1c664725 (DEFAULT_TABLE table=test-table [id=0a46846f2aa44e5c8e2c30ac25ac3048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:42.130024 14318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0866f0e8c2024fcaabe3cf5a1c664725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:42.135334 14567 tablet_bootstrap.cc:492] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa: Bootstrap starting.
I20260807 11:34:42.138541 14567 tablet_bootstrap.cc:654] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:42.139778 14568 tablet_bootstrap.cc:492] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: Bootstrap starting.
I20260807 11:34:42.139874 14567 log.cc:824] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:42.144639 14568 tablet_bootstrap.cc:654] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:42.145563 14570 tablet_bootstrap.cc:492] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: Bootstrap starting.
I20260807 11:34:42.146253 14568 log.cc:824] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:42.149049 14567 tablet_bootstrap.cc:492] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa: No bootstrap required, opened a new log
I20260807 11:34:42.149339 14567 ts_tablet_manager.cc:1397] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260807 11:34:42.149428 14570 tablet_bootstrap.cc:654] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:42.149492 14568 tablet_bootstrap.cc:492] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: No bootstrap required, opened a new log
I20260807 11:34:42.149747 14568 ts_tablet_manager.cc:1397] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260807 11:34:42.150710 14570 log.cc:824] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:42.153932 14570 tablet_bootstrap.cc:492] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: No bootstrap required, opened a new log
I20260807 11:34:42.154217 14570 ts_tablet_manager.cc:1397] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:34:42.157752 14567 raft_consensus.cc:348] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e992cf00a7064ae8b56af666f535120a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36331 } } peers { permanent_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37585 } } peers { permanent_uuid: "a5acb451bd844e7e99c31163119bab6b" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41431 } }
I20260807 11:34:42.158159 14567 raft_consensus.cc:374] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:42.158258 14567 raft_consensus.cc:729] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2de2dd9a6f5464594875b9e2af32aaa, State: Initialized, Role: FOLLOWER
I20260807 11:34:42.158896 14567 consensus_queue.cc:260] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e992cf00a7064ae8b56af666f535120a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36331 } } peers { permanent_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37585 } } peers { permanent_uuid: "a5acb451bd844e7e99c31163119bab6b" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41431 } }
I20260807 11:34:42.160377 14568 raft_consensus.cc:348] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e992cf00a7064ae8b56af666f535120a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36331 } } peers { permanent_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37585 } } peers { permanent_uuid: "a5acb451bd844e7e99c31163119bab6b" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41431 } }
I20260807 11:34:42.160815 14568 raft_consensus.cc:374] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:42.160827 14567 ts_tablet_manager.cc:1428] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260807 11:34:42.160913 14568 raft_consensus.cc:729] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e992cf00a7064ae8b56af666f535120a, State: Initialized, Role: FOLLOWER
I20260807 11:34:42.161525 14568 consensus_queue.cc:260] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e992cf00a7064ae8b56af666f535120a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36331 } } peers { permanent_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37585 } } peers { permanent_uuid: "a5acb451bd844e7e99c31163119bab6b" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41431 } }
I20260807 11:34:42.162703 14546 heartbeater.cc:503] Master 127.24.103.126:45425 was elected leader, sending a full tablet report...
I20260807 11:34:42.163437 14568 ts_tablet_manager.cc:1428] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260807 11:34:42.164947 14570 raft_consensus.cc:348] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e992cf00a7064ae8b56af666f535120a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36331 } } peers { permanent_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37585 } } peers { permanent_uuid: "a5acb451bd844e7e99c31163119bab6b" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41431 } }
I20260807 11:34:42.165458 14570 raft_consensus.cc:374] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:42.165562 14570 raft_consensus.cc:729] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5acb451bd844e7e99c31163119bab6b, State: Initialized, Role: FOLLOWER
I20260807 11:34:42.166162 14570 consensus_queue.cc:260] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e992cf00a7064ae8b56af666f535120a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36331 } } peers { permanent_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37585 } } peers { permanent_uuid: "a5acb451bd844e7e99c31163119bab6b" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41431 } }
I20260807 11:34:42.167804 14394 heartbeater.cc:503] Master 127.24.103.126:45425 was elected leader, sending a full tablet report...
I20260807 11:34:42.169667 14570 ts_tablet_manager.cc:1428] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260807 11:34:42.227167 14574 raft_consensus.cc:482] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:34:42.227624 14574 raft_consensus.cc:504] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e992cf00a7064ae8b56af666f535120a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36331 } } peers { permanent_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37585 } } peers { permanent_uuid: "a5acb451bd844e7e99c31163119bab6b" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41431 } }
I20260807 11:34:42.229745 14574 leader_election.cc:288] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e992cf00a7064ae8b56af666f535120a (127.24.103.67:36331), a5acb451bd844e7e99c31163119bab6b (127.24.103.66:41431)
I20260807 11:34:42.237210 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.230274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:47669 (local address 127.24.103.66:41431)
0807 11:34:42.230467 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:42.230475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:42.231352 (+ 877us) server_negotiation.cc:407] Connection header received
0807 11:34:42.231588 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:42.231594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:42.231690 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:42.231796 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:42.232503 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.233276 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.235407 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.236147 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.236271 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.236319 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:42.236770 (+ 451us) server_negotiation.cc:299] Negotiation successful
0807 11:34:42.236908 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 11:34:42.237241 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.230204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:47371 (local address 127.24.103.67:36331)
0807 11:34:42.230451 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:42.230457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:42.231101 (+ 644us) server_negotiation.cc:407] Connection header received
0807 11:34:42.231322 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:42.231327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:42.231429 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:42.231533 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:42.232305 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.233188 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.235455 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.236353 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.236543 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.236615 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:42.236832 (+ 217us) server_negotiation.cc:299] Negotiation successful
0807 11:34:42.236963 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260807 11:34:42.237336 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.230082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:36331 (local address 127.24.103.65:47371)
0807 11:34:42.230923 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.230947 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:42.231155 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:42.231574 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:42.231579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:42.231606 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:42.232124 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.232137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.233338 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.233347 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.235229 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.235241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.236530 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.236535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.236649 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.236691 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 11:34:42.236789 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 11:34:42.236909 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":86,"thread_start_us":419,"threads_started":1}
I20260807 11:34:42.237584 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.230179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:41431 (local address 127.24.103.65:47669)
0807 11:34:42.231224 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.231255 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:42.231397 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:42.231833 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:42.231836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:42.231858 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:42.232302 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.232314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.233439 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.233443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.235195 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:42.235208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.236358 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.236365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:42.236570 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.236606 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 11:34:42.236705 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 11:34:42.236860 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":32,"thread_start_us":616,"threads_started":1}
I20260807 11:34:42.238538 14496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0866f0e8c2024fcaabe3cf5a1c664725" candidate_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e992cf00a7064ae8b56af666f535120a" is_pre_election: true
I20260807 11:34:42.238732 14341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0866f0e8c2024fcaabe3cf5a1c664725" candidate_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5acb451bd844e7e99c31163119bab6b" is_pre_election: true
I20260807 11:34:42.239317 14496 raft_consensus.cc:2454] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2de2dd9a6f5464594875b9e2af32aaa in term 0.
I20260807 11:34:42.239328 14341 raft_consensus.cc:2454] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2de2dd9a6f5464594875b9e2af32aaa in term 0.
I20260807 11:34:42.240147 14110 leader_election.cc:302] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2de2dd9a6f5464594875b9e2af32aaa, e992cf00a7064ae8b56af666f535120a; no voters:
I20260807 11:34:42.240618 14574 raft_consensus.cc:2781] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:34:42.240775 14574 raft_consensus.cc:482] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:34:42.240893 14574 raft_consensus.cc:3037] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:42.243366 14574 raft_consensus.cc:504] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e992cf00a7064ae8b56af666f535120a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36331 } } peers { permanent_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37585 } } peers { permanent_uuid: "a5acb451bd844e7e99c31163119bab6b" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41431 } }
I20260807 11:34:42.244014 14574 leader_election.cc:288] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [CANDIDATE]: Term 1 election: Requested vote from peers e992cf00a7064ae8b56af666f535120a (127.24.103.67:36331), a5acb451bd844e7e99c31163119bab6b (127.24.103.66:41431)
I20260807 11:34:42.244596 14496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0866f0e8c2024fcaabe3cf5a1c664725" candidate_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e992cf00a7064ae8b56af666f535120a"
I20260807 11:34:42.244580 14341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0866f0e8c2024fcaabe3cf5a1c664725" candidate_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5acb451bd844e7e99c31163119bab6b"
I20260807 11:34:42.244800 14496 raft_consensus.cc:3037] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:42.244809 14341 raft_consensus.cc:3037] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:42.248169 14341 raft_consensus.cc:2454] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2de2dd9a6f5464594875b9e2af32aaa in term 1.
I20260807 11:34:42.248169 14496 raft_consensus.cc:2454] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2de2dd9a6f5464594875b9e2af32aaa in term 1.
I20260807 11:34:42.248653 14108 leader_election.cc:302] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a5acb451bd844e7e99c31163119bab6b, c2de2dd9a6f5464594875b9e2af32aaa, e992cf00a7064ae8b56af666f535120a; no voters:
I20260807 11:34:42.249043 14574 raft_consensus.cc:2781] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:42.249570 14574 raft_consensus.cc:686] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 1 LEADER]: Becoming Leader. State: Replica: c2de2dd9a6f5464594875b9e2af32aaa, State: Running, Role: LEADER
I20260807 11:34:42.250144 14574 consensus_queue.cc:237] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e992cf00a7064ae8b56af666f535120a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36331 } } peers { permanent_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37585 } } peers { permanent_uuid: "a5acb451bd844e7e99c31163119bab6b" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41431 } }
I20260807 11:34:42.255076 14025 catalog_manager.cc:5495] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to c2de2dd9a6f5464594875b9e2af32aaa (127.24.103.65). New cstate: current_term: 1 leader_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e992cf00a7064ae8b56af666f535120a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2de2dd9a6f5464594875b9e2af32aaa" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5acb451bd844e7e99c31163119bab6b" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41431 } health_report { overall_health: UNKNOWN } } }
I20260807 11:34:42.273465 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.267406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.24.103.65:37585)
0807 11:34:42.267591 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:42.267596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:42.267614 (+ 18us) server_negotiation.cc:407] Connection header received
0807 11:34:42.267842 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:42.267846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:42.267965 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:42.268115 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:34:42.268589 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.269349 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.271186 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.271627 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.271743 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.272295 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:34:42.272831 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:34:42.272966 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:42.273193 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 11:34:42.273231 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 11:34:42.281946 14341 raft_consensus.cc:1261] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b [term 1 FOLLOWER]: Refusing update from remote peer c2de2dd9a6f5464594875b9e2af32aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:42.281968 14496 raft_consensus.cc:1261] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a [term 1 FOLLOWER]: Refusing update from remote peer c2de2dd9a6f5464594875b9e2af32aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:42.283178 14579 consensus_queue.cc:1035] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5acb451bd844e7e99c31163119bab6b" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:34:42.283615 14574 consensus_queue.cc:1035] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e992cf00a7064ae8b56af666f535120a" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 11:34:42.284169 14396 tablet.cc:2369] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:34:42.289265 14243 tablet.cc:2369] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:34:42.293150 14547 tablet.cc:2369] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:34:42.297098 14582 mvcc.cc:205] Tried to move back new op lower bound from 7315875767416639488 to 7315875767300849664. Current Snapshot: MvccSnapshot[applied={T|T < 7315875767416639488}]
I20260807 11:34:42.312563 14026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:42.312860 14026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:42.317021 14026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a73dd9728ccd464daa17767cb9621102: Sending DeleteTablet for 3 replicas of tablet 0866f0e8c2024fcaabe3cf5a1c664725
I20260807 11:34:42.318475 14167 tablet_service.cc:1514] Processing DeleteTablet for tablet 0866f0e8c2024fcaabe3cf5a1c664725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:42 UTC) from {username='slave'} at 127.0.0.1:34888
I20260807 11:34:42.318475 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet 0866f0e8c2024fcaabe3cf5a1c664725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:42 UTC) from {username='slave'} at 127.0.0.1:37158
I20260807 11:34:42.318555 14318 tablet_service.cc:1514] Processing DeleteTablet for tablet 0866f0e8c2024fcaabe3cf5a1c664725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:42 UTC) from {username='slave'} at 127.0.0.1:49052
I20260807 11:34:42.320245 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 14090
I20260807 11:34:42.336440 14596 tablet_replica.cc:330] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: stopping tablet replica
I20260807 11:34:42.337059 14596 raft_consensus.cc:2229] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:42.337675 14596 raft_consensus.cc:2258] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:42.340365 14596 ts_tablet_manager.cc:1905] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:42.341754 14594 tablet_replica.cc:330] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: stopping tablet replica
I20260807 11:34:42.342216 14594 raft_consensus.cc:2229] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:42.342779 14594 raft_consensus.cc:2258] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:42.343819 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.322126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.24.103.65:37585)
0807 11:34:42.322372 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:42.322382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:42.333365 (+ 10983us) server_negotiation.cc:407] Connection header received
0807 11:34:42.333967 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:42.333975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:42.334099 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:42.334185 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:42.334961 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.336535 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.338939 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.339720 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.339852 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.341613 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:42.341628 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:42.341646 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:42.342531 (+ 885us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:42.342603 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:42.342631 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:42.342720 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:42.343253 (+ 533us) server_negotiation.cc:299] Negotiation successful
0807 11:34:42.343391 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:34:42.345206 14594 ts_tablet_manager.cc:1905] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:42.345448 14237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:42.352240 14596 ts_tablet_manager.cc:1918] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:42.352468 14596 log.cc:1196] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/0866f0e8c2024fcaabe3cf5a1c664725
I20260807 11:34:42.353223 14596 ts_tablet_manager.cc:1939] T 0866f0e8c2024fcaabe3cf5a1c664725 P a5acb451bd844e7e99c31163119bab6b: Deleting consensus metadata
I20260807 11:34:42.356068 14006 catalog_manager.cc:4841] TS a5acb451bd844e7e99c31163119bab6b (127.24.103.66:41431): tablet 0866f0e8c2024fcaabe3cf5a1c664725 (table test-table [id=0a46846f2aa44e5c8e2c30ac25ac3048]) successfully deleted
I20260807 11:34:42.356925 14594 ts_tablet_manager.cc:1918] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:42.357129 14594 log.cc:1196] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/0866f0e8c2024fcaabe3cf5a1c664725
I20260807 11:34:42.357817 14594 ts_tablet_manager.cc:1939] T 0866f0e8c2024fcaabe3cf5a1c664725 P e992cf00a7064ae8b56af666f535120a: Deleting consensus metadata
I20260807 11:34:42.360230 14008 catalog_manager.cc:4841] TS e992cf00a7064ae8b56af666f535120a (127.24.103.67:36331): tablet 0866f0e8c2024fcaabe3cf5a1c664725 (table test-table [id=0a46846f2aa44e5c8e2c30ac25ac3048]) successfully deleted
I20260807 11:34:42.462805 14595 tablet_replica.cc:330] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa: stopping tablet replica
I20260807 11:34:42.463409 14595 raft_consensus.cc:2229] T 0866f0e8c2024fcaabe3cf5a1c664725 P c2de2dd9a6f5464594875b9e2af32aaa [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:34:42.463778 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 14090
I20260807 11:34:42.487816 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 14246
W20260807 11:34:42.487785 14009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.103.65:37585 (error 108)
I20260807 11:34:42.490114 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.489423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37585 (local address 127.0.0.1:34928)
0807 11:34:42.489672 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.489813 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260807 11:34:42.490406 14009 catalog_manager.cc:4584] TS c2de2dd9a6f5464594875b9e2af32aaa (127.24.103.65:37585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0866f0e8c2024fcaabe3cf5a1c664725: Network error: Client connection negotiation failed: client connection to 127.24.103.65:37585: connect: Connection refused (error 111)
I20260807 11:34:42.495903 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.488816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.24.103.66:41431)
0807 11:34:42.489128 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:42.489135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:42.489161 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:34:42.489339 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:42.489345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:42.489460 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:42.489580 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:42.490145 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.491128 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.492643 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.493083 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.493245 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.494422 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:42.494430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:42.494444 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:42.495204 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:42.495263 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:42.495284 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:42.495396 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:42.495541 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 11:34:42.495635 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 11:34:42.496649 14387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:42.544509 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.543943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37585 (local address 127.0.0.1:34940)
0807 11:34:42.544228 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.544327 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260807 11:34:42.545734 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.545329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37585 (local address 127.0.0.1:34948)
0807 11:34:42.545511 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.545600 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 11:34:42.591780 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 14246
I20260807 11:34:42.606459 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.605836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37585 (local address 127.0.0.1:34960)
0807 11:34:42.606156 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.606257 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 11:34:42.607599 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.607243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37585 (local address 127.0.0.1:34966)
0807 11:34:42.607389 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.607466 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 11:34:42.614730 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 14398
I20260807 11:34:42.621944 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.615676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.24.103.67:36331)
0807 11:34:42.615887 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:42.615895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:42.615916 (+ 21us) server_negotiation.cc:407] Connection header received
0807 11:34:42.615980 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:42.615984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:42.616075 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:42.616185 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:42.616641 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.617520 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.619059 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:42.619474 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:42.619576 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:42.620671 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:42.620680 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:42.620694 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:42.621288 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:42.621337 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:42.621357 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:42.621446 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:42.621591 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 11:34:42.621688 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:34:42.622660 14539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:42.689167 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.688598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37585 (local address 127.0.0.1:34968)
0807 11:34:42.688813 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.688942 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 11:34:42.690505 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:42.690073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37585 (local address 127.0.0.1:34980)
0807 11:34:42.690283 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:42.690370 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.65:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260807 11:34:42.715876 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 14398
I20260807 11:34:42.738050 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 13989
I20260807 11:34:42.739378 14058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:42.853430 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 13989
2026-08-07T11:34:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2960 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:34:42 dist-test-slave-wl2h krb5kdc[14606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:34:42 dist-test-slave-wl2h krb5kdc[14606](info): set up 2 sockets
Aug 07 11:34:42 dist-test-slave-wl2h krb5kdc[14606](info): commencing operation
krb5kdc: starting...
W20260807 11:34:44.132061 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:34:44 dist-test-slave-wl2h krb5kdc[14606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:34:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:34:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:34:44.499589 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:33763
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:44.869285 14619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:44.869598 14619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:44.869695 14619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:44.877136 14619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:34:44.877265 14619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:44.877312 14619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:44.877372 14619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:34:44.877417 14619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:34:44.887588 14619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:33763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:44.888957 14619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:44.890257 14619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:44.902498 14624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:44.902757 14627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:44.902498 14625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:44.903998 14619 server_base.cc:1034] running on GCE node
I20260807 11:34:44.904870 14619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:44.906816 14619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:44.908063 14619 hybrid_clock.cc:648] HybridClock initialized: now 1786102484907984 us; error 62 us; skew 500 ppm
I20260807 11:34:44.908502 14619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:44 dist-test-slave-wl2h krb5kdc[14606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102484, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:44.917519 14619 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:34:44.919265 14619 webserver.cc:460] Webserver started at http://127.24.103.126:41467/ using document root <none> and password file <none>
I20260807 11:34:44.919972 14619 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:44.920080 14619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:44.920360 14619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:44.923575 14619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "e3a35b5ff0324ae780034306c41c3034"
format_stamp: "Formatted at 2026-08-07 11:34:44 on dist-test-slave-wl2h"
server_key: "fb2fcceb26999c4460d51703fe63c625"
server_key_iv: "70257453d28040e1ee578ed8709c8a1a"
server_key_version: "encryptionkey@0"
I20260807 11:34:44.924342 14619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "e3a35b5ff0324ae780034306c41c3034"
format_stamp: "Formatted at 2026-08-07 11:34:44 on dist-test-slave-wl2h"
server_key: "fb2fcceb26999c4460d51703fe63c625"
server_key_iv: "70257453d28040e1ee578ed8709c8a1a"
server_key_version: "encryptionkey@0"
I20260807 11:34:44.929982 14619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:34:44.933991 14636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:44.935583 14619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:34:44.935796 14619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "e3a35b5ff0324ae780034306c41c3034"
format_stamp: "Formatted at 2026-08-07 11:34:44 on dist-test-slave-wl2h"
server_key: "fb2fcceb26999c4460d51703fe63c625"
server_key_iv: "70257453d28040e1ee578ed8709c8a1a"
server_key_version: "encryptionkey@0"
I20260807 11:34:44.936012 14619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:44.955003 14619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:44.960109 14619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:44.960536 14619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:44.982863 14619 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:33763
I20260807 11:34:44.982894 14698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:33763 every 8 connection(s)
I20260807 11:34:44.984468 14619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:34:44.986524 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 14619
I20260807 11:34:44.986759 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:34:44.987426 24989 external_mini_cluster.cc:1428] Setting key d105e6c10cb3b66e4aff3d29d449ec0f
I20260807 11:34:44.989826 14699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:34:45 dist-test-slave-wl2h krb5kdc[14606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:45.005880 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034: Bootstrap starting.
I20260807 11:34:45.009621 14699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:45.010792 14699 log.cc:824] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:45.014997 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034: No bootstrap required, opened a new log
I20260807 11:34:45.015913 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:44.989532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.24.103.126:33763)
0807 11:34:44.990419 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:44.990430 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:44.990474 (+ 44us) server_negotiation.cc:407] Connection header received
0807 11:34:44.992296 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:44.992319 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:44.992727 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:44.993069 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:44.995337 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:44.996774 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:44.998835 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:44.999674 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:44.999855 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:45.006290 (+ 6435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:45.006300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:45.006317 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:45.006363 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:45.010091 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:45.011455 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:45.011700 (+ 245us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:45.011715 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:45.012044 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:45.012578 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:45.012585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:45.012588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:45.013195 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:45.013531 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:45.014035 (+ 504us) server_negotiation.cc:299] Negotiation successful
0807 11:34:45.014554 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":415,"threads_started":1}
I20260807 11:34:45.026906 14699 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a35b5ff0324ae780034306c41c3034" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33763 } }
I20260807 11:34:45.027577 14699 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:45.027697 14699 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a35b5ff0324ae780034306c41c3034, State: Initialized, Role: FOLLOWER
I20260807 11:34:45.028390 14699 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a35b5ff0324ae780034306c41c3034" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33763 } }
I20260807 11:34:45.028671 14699 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:34:45.028831 14699 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:34:45.029014 14699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:45.031565 14699 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a35b5ff0324ae780034306c41c3034" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33763 } }
I20260807 11:34:45.032146 14699 leader_election.cc:302] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3a35b5ff0324ae780034306c41c3034; no voters:
I20260807 11:34:45.032908 14699 leader_election.cc:288] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:34:45.033094 14706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:45.034310 14706 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [term 1 LEADER]: Becoming Leader. State: Replica: e3a35b5ff0324ae780034306c41c3034, State: Running, Role: LEADER
I20260807 11:34:45.034925 14706 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a35b5ff0324ae780034306c41c3034" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33763 } }
I20260807 11:34:45.035408 14699 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:34:45.038282 14710 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3a35b5ff0324ae780034306c41c3034. Latest consensus state: current_term: 1 leader_uuid: "e3a35b5ff0324ae780034306c41c3034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a35b5ff0324ae780034306c41c3034" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33763 } } }
I20260807 11:34:45.038383 14708 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3a35b5ff0324ae780034306c41c3034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a35b5ff0324ae780034306c41c3034" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 33763 } } }
I20260807 11:34:45.038913 14710 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:45.038909 14708 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:45.040570 14714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:34:45.046540 14714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:34:45.064567 14714 catalog_manager.cc:1347] Generated new cluster ID: 9e0f803a307745ceb7afb41c24d1a533
I20260807 11:34:45.064713 14714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:34:45.082086 14714 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:34:45.084194 14714 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:34:45.102234 14714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034: Generated new TSK 0
I20260807 11:34:45.103367 14714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.103.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
Entry for principal kudu/127.24.103.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab.
I20260807 11:34:45.192430 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33763
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:45.560081 14734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:45.560407 14734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:45.560572 14734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:45.568195 14734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:45.568316 14734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:45.568487 14734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:34:45.579068 14734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.65.keytab
--principal=kudu/127.24.103.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:45.580420 14734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:45.581756 14734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:45.595669 14742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:45.595635 14746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:45.595609 14744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:45.596884 14734 server_base.cc:1034] running on GCE node
I20260807 11:34:45.597523 14734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:45.598570 14734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:45.599815 14734 hybrid_clock.cc:648] HybridClock initialized: now 1786102485599746 us; error 59 us; skew 500 ppm
I20260807 11:34:45.600211 14734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:45 dist-test-slave-wl2h krb5kdc[14606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102485, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:45.604434 14734 init.cc:377] Logged in from keytab as kudu/127.24.103.65@KRBTEST.COM (short username kudu)
I20260807 11:34:45.606244 14734 webserver.cc:460] Webserver started at http://127.24.103.65:39623/ using document root <none> and password file <none>
I20260807 11:34:45.607107 14734 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:45.607278 14734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:45.607671 14734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:45.610922 14734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "bc77d02acdcf47feaba8032556a8edcc"
format_stamp: "Formatted at 2026-08-07 11:34:45 on dist-test-slave-wl2h"
server_key: "8aad609aaab9065fe28202eba91e3e56"
server_key_iv: "0a2b15b01fe9ba1a7a3149a9c217a348"
server_key_version: "encryptionkey@0"
I20260807 11:34:45.611871 14734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "bc77d02acdcf47feaba8032556a8edcc"
format_stamp: "Formatted at 2026-08-07 11:34:45 on dist-test-slave-wl2h"
server_key: "8aad609aaab9065fe28202eba91e3e56"
server_key_iv: "0a2b15b01fe9ba1a7a3149a9c217a348"
server_key_version: "encryptionkey@0"
I20260807 11:34:45.617637 14734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:34:45.621627 14754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:45.623203 14734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 11:34:45.623389 14734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "bc77d02acdcf47feaba8032556a8edcc"
format_stamp: "Formatted at 2026-08-07 11:34:45 on dist-test-slave-wl2h"
server_key: "8aad609aaab9065fe28202eba91e3e56"
server_key_iv: "0a2b15b01fe9ba1a7a3149a9c217a348"
server_key_version: "encryptionkey@0"
I20260807 11:34:45.623592 14734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:45.648506 14734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:45.653304 14734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:45.653622 14734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:45.654747 14734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:45.656327 14734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:45.656435 14734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:45.656587 14734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:45.656666 14734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:45.692364 14734 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:37531
I20260807 11:34:45.692451 14882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:37531 every 8 connection(s)
I20260807 11:34:45.693698 14734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:34:45.700057 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 14734
I20260807 11:34:45.700265 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:34:45.701009 24989 external_mini_cluster.cc:1428] Setting key a0874ab080932c75c8a828c18334147c
Aug 07 11:34:45 dist-test-slave-wl2h krb5kdc[14606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102485, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.65@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:45.717962 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:45.696460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:47675 (local address 127.24.103.126:33763)
0807 11:34:45.696918 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:45.696923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:45.698053 (+ 1130us) server_negotiation.cc:407] Connection header received
0807 11:34:45.701446 (+ 3393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:45.701452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:45.701525 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:45.701636 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:45.703743 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:45.704571 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:45.707364 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:45.707775 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:45.707880 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:45.712392 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:45.712397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:45.712400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:45.712432 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:45.714922 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:45.715669 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:45.715679 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:45.715681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:45.715795 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:45.716236 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:45.716240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:45.716248 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:45.716502 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:45.716665 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:45.717363 (+ 698us) server_negotiation.cc:299] Negotiation successful
0807 11:34:45.717590 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
I20260807 11:34:45.718986 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:45.696814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33763 (local address 127.24.103.65:47675)
0807 11:34:45.697819 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:45.697888 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:45.701110 (+ 3222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:45.701916 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:45.701926 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:45.702657 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:45.703506 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:45.703535 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:45.704763 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:45.704772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:45.707205 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:45.707226 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:45.708027 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:45.708038 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:45.708398 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:45.709187 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:34:45.709232 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:45.712156 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:45.715081 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:45.715096 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:45.715113 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:45.715538 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:45.715919 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:45.715924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:45.715928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:45.716119 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:45.716645 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:45.716653 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:45.716983 (+ 330us) client_negotiation.cc:769] Sending connection context
0807 11:34:45.717244 (+ 261us) client_negotiation.cc:240] Negotiation successful
0807 11:34:45.717817 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":373,"threads_started":1}
I20260807 11:34:45.720700 14883 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33763
I20260807 11:34:45.720995 14883 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:45.721715 14883 heartbeater.cc:511] Master 127.24.103.126:33763 requested a full tablet report, sending...
I20260807 11:34:45.724260 14656 ts_manager.cc:194] Registered new tserver with Master: bc77d02acdcf47feaba8032556a8edcc (127.24.103.65:37531)
I20260807 11:34:45.726722 14656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.65@KRBTEST.COM'} at 127.24.103.65:47675
WARNING: no policy specified for kudu/127.24.103.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
Entry for principal kudu/127.24.103.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab.
I20260807 11:34:45.779568 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33763
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:46.133299 14890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:46.133594 14890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:46.133692 14890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:46.141088 14890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:46.141199 14890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:46.141323 14890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:34:46.151121 14890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.66.keytab
--principal=kudu/127.24.103.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:46.152535 14890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:46.153811 14890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:46.167282 14895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:46.167318 14896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:46.167344 14899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:46.168676 14890 server_base.cc:1034] running on GCE node
I20260807 11:34:46.169360 14890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:46.170394 14890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:46.171630 14890 hybrid_clock.cc:648] HybridClock initialized: now 1786102486171572 us; error 46 us; skew 500 ppm
I20260807 11:34:46.171959 14890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:46 dist-test-slave-wl2h krb5kdc[14606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102486, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:46.175386 14890 init.cc:377] Logged in from keytab as kudu/127.24.103.66@KRBTEST.COM (short username kudu)
I20260807 11:34:46.176816 14890 webserver.cc:460] Webserver started at http://127.24.103.66:46837/ using document root <none> and password file <none>
I20260807 11:34:46.177477 14890 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:46.177579 14890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:46.177837 14890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:46.180794 14890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "d917e277b2254658ae274dab901cf410"
format_stamp: "Formatted at 2026-08-07 11:34:46 on dist-test-slave-wl2h"
server_key: "6ad8c5ad9f6edf4f4d7b1177f8b688a7"
server_key_iv: "9c683b299f0a72de4b774dfefd9e8cd1"
server_key_version: "encryptionkey@0"
I20260807 11:34:46.181546 14890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "d917e277b2254658ae274dab901cf410"
format_stamp: "Formatted at 2026-08-07 11:34:46 on dist-test-slave-wl2h"
server_key: "6ad8c5ad9f6edf4f4d7b1177f8b688a7"
server_key_iv: "9c683b299f0a72de4b774dfefd9e8cd1"
server_key_version: "encryptionkey@0"
I20260807 11:34:46.186965 14890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:34:46.190690 14907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:46.192354 14890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:34:46.192575 14890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "d917e277b2254658ae274dab901cf410"
format_stamp: "Formatted at 2026-08-07 11:34:46 on dist-test-slave-wl2h"
server_key: "6ad8c5ad9f6edf4f4d7b1177f8b688a7"
server_key_iv: "9c683b299f0a72de4b774dfefd9e8cd1"
server_key_version: "encryptionkey@0"
I20260807 11:34:46.192802 14890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:46.253161 14890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:46.257974 14890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:46.258258 14890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:46.259433 14890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:46.260952 14890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:46.261035 14890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:46.261138 14890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:46.261193 14890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:46.298421 14890 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:39045
I20260807 11:34:46.298470 15031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:39045 every 8 connection(s)
I20260807 11:34:46.299986 14890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:34:46.305351 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 14890
I20260807 11:34:46.305583 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:34:46.306172 24989 external_mini_cluster.cc:1428] Setting key 40f2ef87b544f56567513b5dd29ca28d
Aug 07 11:34:46 dist-test-slave-wl2h krb5kdc[14606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102486, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.66@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:46.323444 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:46.302825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:39247 (local address 127.24.103.126:33763)
0807 11:34:46.303421 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:46.303426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:46.304450 (+ 1024us) server_negotiation.cc:407] Connection header received
0807 11:34:46.307077 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:46.307084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:46.307179 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:46.307328 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:46.309037 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.309880 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.312477 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.312947 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.313062 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:46.317502 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:46.317507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:46.317510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:46.317545 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:46.320328 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:46.321075 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:46.321080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:46.321082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:46.321210 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:46.321644 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:46.321648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:46.321650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:46.321935 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:46.322091 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:46.322802 (+ 711us) server_negotiation.cc:299] Negotiation successful
0807 11:34:46.323058 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":358,"threads_started":1}
I20260807 11:34:46.324458 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:46.303489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33763 (local address 127.24.103.66:39247)
0807 11:34:46.304264 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:46.304316 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:46.306748 (+ 2432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:46.307558 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:46.307566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:46.308079 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:46.308835 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:46.308865 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.310071 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.310079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:46.312309 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:46.312327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.313208 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.313218 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:46.313560 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:46.314395 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:34:46.314431 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:46.317251 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:46.320478 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:46.320487 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:46.320505 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:46.320938 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:46.321319 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:46.321324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:46.321328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:46.321520 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:46.322065 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:46.322073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:46.322432 (+ 359us) client_negotiation.cc:769] Sending connection context
0807 11:34:46.322696 (+ 264us) client_negotiation.cc:240] Negotiation successful
0807 11:34:46.323280 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":305,"threads_started":1}
I20260807 11:34:46.325924 15032 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33763
I20260807 11:34:46.326243 15032 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:46.326972 15032 heartbeater.cc:511] Master 127.24.103.126:33763 requested a full tablet report, sending...
I20260807 11:34:46.328588 14656 ts_manager.cc:194] Registered new tserver with Master: d917e277b2254658ae274dab901cf410 (127.24.103.66:39045)
I20260807 11:34:46.329913 14656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.66@KRBTEST.COM'} at 127.24.103.66:39247
WARNING: no policy specified for kudu/127.24.103.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
Entry for principal kudu/127.24.103.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab.
I20260807 11:34:46.383503 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33763
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:34:46.730388 14883 heartbeater.cc:503] Master 127.24.103.126:33763 was elected leader, sending a full tablet report...
W20260807 11:34:46.734279 15041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:46.734633 15041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:46.734788 15041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:46.742364 15041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:46.742518 15041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:46.742687 15041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:34:46.752628 15041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.67.keytab
--principal=kudu/127.24.103.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:33763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:46.754072 15041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:46.755426 15041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:46.769505 15051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:46.769562 15052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:46.769618 15054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:46.775830 15041 server_base.cc:1034] running on GCE node
I20260807 11:34:46.776587 15041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:46.777750 15041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:46.778995 15041 hybrid_clock.cc:648] HybridClock initialized: now 1786102486778924 us; error 62 us; skew 500 ppm
I20260807 11:34:46.779443 15041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:46 dist-test-slave-wl2h krb5kdc[14606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102486, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:46.783202 15041 init.cc:377] Logged in from keytab as kudu/127.24.103.67@KRBTEST.COM (short username kudu)
I20260807 11:34:46.784878 15041 webserver.cc:460] Webserver started at http://127.24.103.67:36625/ using document root <none> and password file <none>
I20260807 11:34:46.785596 15041 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:46.785702 15041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:46.786041 15041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:46.789146 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "31327cf436384f9293bef79a3d5a90b2"
format_stamp: "Formatted at 2026-08-07 11:34:46 on dist-test-slave-wl2h"
server_key: "274589684e5c9f7f29b8aaede7c77f09"
server_key_iv: "6df344ff59e16862c6b45d4093e4b06d"
server_key_version: "encryptionkey@0"
I20260807 11:34:46.789912 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "31327cf436384f9293bef79a3d5a90b2"
format_stamp: "Formatted at 2026-08-07 11:34:46 on dist-test-slave-wl2h"
server_key: "274589684e5c9f7f29b8aaede7c77f09"
server_key_iv: "6df344ff59e16862c6b45d4093e4b06d"
server_key_version: "encryptionkey@0"
I20260807 11:34:46.795637 15041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 11:34:46.799782 15063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:46.801251 15041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:34:46.801409 15041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "31327cf436384f9293bef79a3d5a90b2"
format_stamp: "Formatted at 2026-08-07 11:34:46 on dist-test-slave-wl2h"
server_key: "274589684e5c9f7f29b8aaede7c77f09"
server_key_iv: "6df344ff59e16862c6b45d4093e4b06d"
server_key_version: "encryptionkey@0"
I20260807 11:34:46.801607 15041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:46.826062 15041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:46.831382 15041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:46.831676 15041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:46.832729 15041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:46.834204 15041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:46.834285 15041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:46.834379 15041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:46.834429 15041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:46.871052 15041 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:35423
I20260807 11:34:46.871194 15196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:35423 every 8 connection(s)
I20260807 11:34:46.872429 15041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:34:46.878017 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 15041
I20260807 11:34:46.878309 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:34:46.878832 24989 external_mini_cluster.cc:1428] Setting key 0d6fa3426476b555039280c7cded5523
Aug 07 11:34:46 dist-test-slave-wl2h krb5kdc[14606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102486, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.67@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:34:46.894342 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:46.875180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:46903 (local address 127.24.103.126:33763)
0807 11:34:46.875714 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:46.875718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:46.876567 (+ 849us) server_negotiation.cc:407] Connection header received
0807 11:34:46.878849 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:46.878855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:46.879004 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:46.879166 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:46.880897 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.881734 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.883379 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.883802 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.883940 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:46.887825 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:46.887830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:46.887834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:46.887868 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:46.890725 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:46.891682 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:46.891686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:46.891689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:46.891821 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:46.892325 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:46.892329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:46.892331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:46.892625 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:46.892820 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:46.893593 (+ 773us) server_negotiation.cc:299] Negotiation successful
0807 11:34:46.893860 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":278,"threads_started":1}
I20260807 11:34:46.895439 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:46.875651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:33763 (local address 127.24.103.67:46903)
0807 11:34:46.876408 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:46.876448 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:46.878273 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:46.879468 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:46.879477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:46.880087 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:46.880727 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:46.880744 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.881898 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.881902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:46.883234 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:46.883247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.884048 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.884056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:46.884301 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:46.885059 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:34:46.885088 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:46.887580 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:46.890914 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:46.890924 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:46.890946 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:46.891524 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:46.891947 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:34:46.891952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:34:46.891956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:34:46.892186 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:34:46.892788 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:46.892796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:46.893178 (+ 382us) client_negotiation.cc:769] Sending connection context
0807 11:34:46.893472 (+ 294us) client_negotiation.cc:240] Negotiation successful
0807 11:34:46.894073 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":303,"threads_started":1}
I20260807 11:34:46.896904 15197 heartbeater.cc:347] Connected to a master server at 127.24.103.126:33763
I20260807 11:34:46.897293 15197 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:46.898157 15197 heartbeater.cc:511] Master 127.24.103.126:33763 requested a full tablet report, sending...
I20260807 11:34:46.899863 14656 ts_manager.cc:194] Registered new tserver with Master: 31327cf436384f9293bef79a3d5a90b2 (127.24.103.67:35423)
I20260807 11:34:46.900767 14656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.103.67@KRBTEST.COM'} at 127.24.103.67:46903
I20260807 11:34:46.907249 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:34:46.926709 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:46.911973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.24.103.126:33763)
0807 11:34:46.912263 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:46.912272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:46.912645 (+ 373us) server_negotiation.cc:407] Connection header received
0807 11:34:46.912871 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:46.912876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:46.912962 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:46.913077 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:46.914798 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.915685 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.917125 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.917584 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.917741 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:46.920623 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:46.920629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:46.920632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:46.920672 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:46.923732 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:46.924490 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:46.924496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:46.924500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:46.924641 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:46.925066 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:46.925072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:46.925075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:46.925454 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:46.925720 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:46.926106 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 11:34:46.926374 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 11:34:46.931612 14656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:34:46.939687 14656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:34:46.981721 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:46.953522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:39045 (local address 127.0.0.1:48984)
0807 11:34:46.968786 (+ 15264us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:46.968808 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:46.968953 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:46.969765 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:46.969775 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:46.969822 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:46.970404 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:46.970430 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.973548 (+ 3118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.973558 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:46.977184 (+ 3626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:46.977212 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.980415 (+ 3203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.980426 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:46.980837 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:46.980899 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 11:34:46.981013 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 11:34:46.981194 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15124,"mutex_wait_us":33,"thread_start_us":14885,"threads_started":1}
I20260807 11:34:46.982184 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:46.953758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.24.103.65:37531)
0807 11:34:46.969668 (+ 15910us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:46.969677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:46.969707 (+ 30us) server_negotiation.cc:407] Connection header received
0807 11:34:46.969811 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:46.969819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:46.970076 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:46.970420 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:46.971978 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.974228 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.977816 (+ 3588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.979991 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.981107 (+ 1116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:46.981521 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:46.981630 (+ 109us) server_negotiation.cc:299] Negotiation successful
0807 11:34:46.981827 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15714,"thread_start_us":15553,"threads_started":1}
I20260807 11:34:46.982911 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:46.953431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37531 (local address 127.0.0.1:58776)
0807 11:34:46.967232 (+ 13801us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:46.967298 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:46.967584 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:46.970741 (+ 3157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:46.970749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:46.970788 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:46.971722 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:46.971741 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.974438 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.974447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:46.977504 (+ 3057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:46.977528 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.980362 (+ 2834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.980369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:46.980706 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:46.980773 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 11:34:46.980906 (+ 133us) client_negotiation.cc:240] Negotiation successful
0807 11:34:46.982495 (+ 1589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13375,"mutex_wait_us":1363,"thread_start_us":15048,"threads_started":1}
I20260807 11:34:46.983105 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:46.953760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.24.103.66:39045)
0807 11:34:46.968370 (+ 14610us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:46.968384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:46.969088 (+ 704us) server_negotiation.cc:407] Connection header received
0807 11:34:46.969228 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:46.969239 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:46.969504 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:46.970712 (+ 1208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:46.971069 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.973181 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.977546 (+ 4365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.980035 (+ 2489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.982260 (+ 2225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:46.982463 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:46.982556 (+ 93us) server_negotiation.cc:299] Negotiation successful
0807 11:34:46.982763 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14379,"thread_start_us":14238,"threads_started":1}
I20260807 11:34:46.987149 14962 tablet_service.cc:1467] Processing CreateTablet for tablet 1f95910592184a25bf374f31a529feae (DEFAULT_TABLE table=test-table [id=c453308f46144145bceca43712db8760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:46.988862 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f95910592184a25bf374f31a529feae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:46.991664 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:46.969718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.24.103.67:35423)
0807 11:34:46.980622 (+ 10904us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:46.980632 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:46.981602 (+ 970us) server_negotiation.cc:407] Connection header received
0807 11:34:46.982486 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:46.982496 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:46.982813 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:46.983338 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:46.983954 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.985670 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.988182 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.989641 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.989779 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:46.990001 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:46.991147 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0807 11:34:46.991334 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10687,"thread_start_us":10930,"threads_started":1}
I20260807 11:34:46.990975 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:46.969606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:35423 (local address 127.0.0.1:33236)
0807 11:34:46.981442 (+ 11836us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:46.981468 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:46.981745 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:46.983337 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:46.983342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:46.983368 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:46.983750 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:46.983761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.985826 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.985835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:46.987981 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:46.987998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:46.990161 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:46.990167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:46.990347 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:46.990381 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 11:34:46.990467 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 11:34:46.990629 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11672,"thread_start_us":11391,"threads_started":1}
I20260807 11:34:46.993283 14807 tablet_service.cc:1467] Processing CreateTablet for tablet 1f95910592184a25bf374f31a529feae (DEFAULT_TABLE table=test-table [id=c453308f46144145bceca43712db8760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:46.994922 14807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f95910592184a25bf374f31a529feae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:46.996315 15116 tablet_service.cc:1467] Processing CreateTablet for tablet 1f95910592184a25bf374f31a529feae (DEFAULT_TABLE table=test-table [id=c453308f46144145bceca43712db8760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:46.997725 15116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f95910592184a25bf374f31a529feae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:47.010126 15218 tablet_bootstrap.cc:492] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: Bootstrap starting.
I20260807 11:34:47.015343 15218 tablet_bootstrap.cc:654] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:47.016744 15218 log.cc:824] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:47.016835 15220 tablet_bootstrap.cc:492] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: Bootstrap starting.
I20260807 11:34:47.017540 15219 tablet_bootstrap.cc:492] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: Bootstrap starting.
I20260807 11:34:47.020125 15218 tablet_bootstrap.cc:492] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: No bootstrap required, opened a new log
I20260807 11:34:47.020421 15218 ts_tablet_manager.cc:1397] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260807 11:34:47.020670 15220 tablet_bootstrap.cc:654] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:47.021272 15219 tablet_bootstrap.cc:654] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:47.022017 15220 log.cc:824] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:47.022521 15219 log.cc:824] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:47.026476 15219 tablet_bootstrap.cc:492] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: No bootstrap required, opened a new log
I20260807 11:34:47.026476 15220 tablet_bootstrap.cc:492] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: No bootstrap required, opened a new log
I20260807 11:34:47.026779 15219 ts_tablet_manager.cc:1397] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260807 11:34:47.026779 15220 ts_tablet_manager.cc:1397] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260807 11:34:47.028684 15218 raft_consensus.cc:348] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc77d02acdcf47feaba8032556a8edcc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37531 } } peers { permanent_uuid: "d917e277b2254658ae274dab901cf410" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39045 } } peers { permanent_uuid: "31327cf436384f9293bef79a3d5a90b2" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35423 } }
I20260807 11:34:47.029120 15218 raft_consensus.cc:374] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:47.029248 15218 raft_consensus.cc:729] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d917e277b2254658ae274dab901cf410, State: Initialized, Role: FOLLOWER
I20260807 11:34:47.029925 15218 consensus_queue.cc:260] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc77d02acdcf47feaba8032556a8edcc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37531 } } peers { permanent_uuid: "d917e277b2254658ae274dab901cf410" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39045 } } peers { permanent_uuid: "31327cf436384f9293bef79a3d5a90b2" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35423 } }
I20260807 11:34:47.031193 15032 heartbeater.cc:503] Master 127.24.103.126:33763 was elected leader, sending a full tablet report...
I20260807 11:34:47.032845 15218 ts_tablet_manager.cc:1428] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260807 11:34:47.036492 15220 raft_consensus.cc:348] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc77d02acdcf47feaba8032556a8edcc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37531 } } peers { permanent_uuid: "d917e277b2254658ae274dab901cf410" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39045 } } peers { permanent_uuid: "31327cf436384f9293bef79a3d5a90b2" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35423 } }
I20260807 11:34:47.037001 15220 raft_consensus.cc:374] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:47.037114 15220 raft_consensus.cc:729] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31327cf436384f9293bef79a3d5a90b2, State: Initialized, Role: FOLLOWER
I20260807 11:34:47.037832 15220 consensus_queue.cc:260] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc77d02acdcf47feaba8032556a8edcc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37531 } } peers { permanent_uuid: "d917e277b2254658ae274dab901cf410" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39045 } } peers { permanent_uuid: "31327cf436384f9293bef79a3d5a90b2" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35423 } }
I20260807 11:34:47.038404 15219 raft_consensus.cc:348] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc77d02acdcf47feaba8032556a8edcc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37531 } } peers { permanent_uuid: "d917e277b2254658ae274dab901cf410" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39045 } } peers { permanent_uuid: "31327cf436384f9293bef79a3d5a90b2" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35423 } }
I20260807 11:34:47.038883 15219 raft_consensus.cc:374] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:47.039026 15219 raft_consensus.cc:729] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc77d02acdcf47feaba8032556a8edcc, State: Initialized, Role: FOLLOWER
I20260807 11:34:47.039060 15197 heartbeater.cc:503] Master 127.24.103.126:33763 was elected leader, sending a full tablet report...
I20260807 11:34:47.039796 15220 ts_tablet_manager.cc:1428] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260807 11:34:47.039701 15219 consensus_queue.cc:260] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc77d02acdcf47feaba8032556a8edcc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37531 } } peers { permanent_uuid: "d917e277b2254658ae274dab901cf410" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39045 } } peers { permanent_uuid: "31327cf436384f9293bef79a3d5a90b2" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35423 } }
I20260807 11:34:47.041707 15219 ts_tablet_manager.cc:1428] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260807 11:34:47.052515 15034 tablet.cc:2369] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 11:34:47.124460 15198 tablet.cc:2369] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:34:47.154701 15224 raft_consensus.cc:482] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:34:47.155149 15224 raft_consensus.cc:504] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc77d02acdcf47feaba8032556a8edcc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37531 } } peers { permanent_uuid: "d917e277b2254658ae274dab901cf410" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39045 } } peers { permanent_uuid: "31327cf436384f9293bef79a3d5a90b2" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35423 } }
I20260807 11:34:47.156932 15224 leader_election.cc:288] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc77d02acdcf47feaba8032556a8edcc (127.24.103.65:37531), 31327cf436384f9293bef79a3d5a90b2 (127.24.103.67:35423)
I20260807 11:34:47.165442 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:47.158246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:48811 (local address 127.24.103.67:35423)
0807 11:34:47.158483 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:47.158489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:47.158639 (+ 150us) server_negotiation.cc:407] Connection header received
0807 11:34:47.158755 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:47.158759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:47.158843 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:47.158942 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:47.159687 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.160777 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.163427 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.164324 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.164451 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:47.164506 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:47.164962 (+ 456us) server_negotiation.cc:299] Negotiation successful
0807 11:34:47.165159 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 11:34:47.165510 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:47.157392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:52071 (local address 127.24.103.65:37531)
0807 11:34:47.157648 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:47.157657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:47.157975 (+ 318us) server_negotiation.cc:407] Connection header received
0807 11:34:47.158254 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:47.158259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:47.158346 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:47.158456 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:47.159381 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.160502 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.163397 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.164529 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.164715 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:47.164789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:47.165078 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 11:34:47.165273 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:34:47.165547 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:47.157891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:35423 (local address 127.24.103.66:48811)
0807 11:34:47.158538 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:47.158552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:47.158659 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:47.158979 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:47.158982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:47.159006 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:47.159477 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:47.159487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.160912 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.160917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:47.163221 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:47.163230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.164551 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.164559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:47.164776 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:47.164816 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 11:34:47.164906 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 11:34:47.165025 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":10,"thread_start_us":330,"threads_started":1}
I20260807 11:34:47.165642 15227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:47.157153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:37531 (local address 127.24.103.66:52071)
0807 11:34:47.157845 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:47.157861 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:47.158042 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:47.158710 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:47.158717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:47.158746 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:47.159223 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:47.159235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.160642 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.160649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:47.163202 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:47.163223 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.164750 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.164755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:47.164876 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:47.164933 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 11:34:47.165028 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 11:34:47.165158 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":94,"thread_start_us":354,"threads_started":1}
I20260807 11:34:47.166885 15140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f95910592184a25bf374f31a529feae" candidate_uuid: "d917e277b2254658ae274dab901cf410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31327cf436384f9293bef79a3d5a90b2" is_pre_election: true
I20260807 11:34:47.166797 14829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f95910592184a25bf374f31a529feae" candidate_uuid: "d917e277b2254658ae274dab901cf410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc77d02acdcf47feaba8032556a8edcc" is_pre_election: true
I20260807 11:34:47.167503 15140 raft_consensus.cc:2454] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d917e277b2254658ae274dab901cf410 in term 0.
I20260807 11:34:47.167503 14829 raft_consensus.cc:2454] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d917e277b2254658ae274dab901cf410 in term 0.
I20260807 11:34:47.168330 14908 leader_election.cc:302] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31327cf436384f9293bef79a3d5a90b2, d917e277b2254658ae274dab901cf410; no voters:
I20260807 11:34:47.168782 15224 raft_consensus.cc:2781] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:34:47.168955 15224 raft_consensus.cc:482] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:34:47.169119 15224 raft_consensus.cc:3037] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:47.171478 15224 raft_consensus.cc:504] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc77d02acdcf47feaba8032556a8edcc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37531 } } peers { permanent_uuid: "d917e277b2254658ae274dab901cf410" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39045 } } peers { permanent_uuid: "31327cf436384f9293bef79a3d5a90b2" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35423 } }
I20260807 11:34:47.172089 15224 leader_election.cc:288] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [CANDIDATE]: Term 1 election: Requested vote from peers bc77d02acdcf47feaba8032556a8edcc (127.24.103.65:37531), 31327cf436384f9293bef79a3d5a90b2 (127.24.103.67:35423)
I20260807 11:34:47.172855 14829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f95910592184a25bf374f31a529feae" candidate_uuid: "d917e277b2254658ae274dab901cf410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc77d02acdcf47feaba8032556a8edcc"
I20260807 11:34:47.172943 15140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f95910592184a25bf374f31a529feae" candidate_uuid: "d917e277b2254658ae274dab901cf410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31327cf436384f9293bef79a3d5a90b2"
I20260807 11:34:47.173101 14829 raft_consensus.cc:3037] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:47.173240 15140 raft_consensus.cc:3037] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:47.175575 15140 raft_consensus.cc:2454] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d917e277b2254658ae274dab901cf410 in term 1.
I20260807 11:34:47.175755 14829 raft_consensus.cc:2454] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d917e277b2254658ae274dab901cf410 in term 1.
I20260807 11:34:47.176041 14908 leader_election.cc:302] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31327cf436384f9293bef79a3d5a90b2, d917e277b2254658ae274dab901cf410; no voters:
I20260807 11:34:47.176374 15224 raft_consensus.cc:2781] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:47.176889 15224 raft_consensus.cc:686] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 1 LEADER]: Becoming Leader. State: Replica: d917e277b2254658ae274dab901cf410, State: Running, Role: LEADER
I20260807 11:34:47.177455 15224 consensus_queue.cc:237] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc77d02acdcf47feaba8032556a8edcc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37531 } } peers { permanent_uuid: "d917e277b2254658ae274dab901cf410" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39045 } } peers { permanent_uuid: "31327cf436384f9293bef79a3d5a90b2" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35423 } }
I20260807 11:34:47.182960 14656 catalog_manager.cc:5495] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 reported cstate change: term changed from 0 to 1, leader changed from <none> to d917e277b2254658ae274dab901cf410 (127.24.103.66). New cstate: current_term: 1 leader_uuid: "d917e277b2254658ae274dab901cf410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc77d02acdcf47feaba8032556a8edcc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d917e277b2254658ae274dab901cf410" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 39045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31327cf436384f9293bef79a3d5a90b2" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35423 } health_report { overall_health: UNKNOWN } } }
W20260807 11:34:47.196350 14884 tablet.cc:2369] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:34:47.206238 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:47.200114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.24.103.66:39045)
0807 11:34:47.200363 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:47.200370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:47.200471 (+ 101us) server_negotiation.cc:407] Connection header received
0807 11:34:47.200728 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:47.200732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:47.200835 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:47.200991 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:34:47.201419 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.202209 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.203840 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.204264 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.204407 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:47.204927 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:34:47.205566 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:34:47.205724 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:47.205817 (+ 93us) server_negotiation.cc:299] Negotiation successful
0807 11:34:47.205910 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 11:34:47.215471 14829 raft_consensus.cc:1261] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc [term 1 FOLLOWER]: Refusing update from remote peer d917e277b2254658ae274dab901cf410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:47.215634 15140 raft_consensus.cc:1261] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2 [term 1 FOLLOWER]: Refusing update from remote peer d917e277b2254658ae274dab901cf410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:47.216622 15229 consensus_queue.cc:1035] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc77d02acdcf47feaba8032556a8edcc" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 37531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:34:47.217008 15224 consensus_queue.cc:1035] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31327cf436384f9293bef79a3d5a90b2" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:34:47.227735 15232 mvcc.cc:205] Tried to move back new op lower bound from 7315875787623010304 to 7315875787483627520. Current Snapshot: MvccSnapshot[applied={T|T < 7315875787623010304}]
I20260807 11:34:47.229950 15234 mvcc.cc:205] Tried to move back new op lower bound from 7315875787623010304 to 7315875787483627520. Current Snapshot: MvccSnapshot[applied={T|T < 7315875787623010304}]
I20260807 11:34:47.250064 14655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:47.250396 14655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:47.254424 14655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3a35b5ff0324ae780034306c41c3034: Sending DeleteTablet for 3 replicas of tablet 1f95910592184a25bf374f31a529feae
I20260807 11:34:47.255600 14807 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f95910592184a25bf374f31a529feae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:47 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:58776
I20260807 11:34:47.255720 15116 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f95910592184a25bf374f31a529feae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:47 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:33236
I20260807 11:34:47.256031 14962 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f95910592184a25bf374f31a529feae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:47 UTC) from {username='kudu', principal='kudu/127.24.103.126@KRBTEST.COM'} at 127.0.0.1:48984
I20260807 11:34:47.260556 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 14734
I20260807 11:34:47.268613 15244 tablet_replica.cc:330] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: stopping tablet replica
I20260807 11:34:47.269675 15244 raft_consensus.cc:2229] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:34:47.270882 15244 raft_consensus.cc:2258] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:47.273002 15243 tablet_replica.cc:330] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: stopping tablet replica
I20260807 11:34:47.273693 15243 raft_consensus.cc:2229] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:47.274483 15243 raft_consensus.cc:2258] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:47.275269 15242 tablet_replica.cc:330] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: stopping tablet replica
I20260807 11:34:47.275820 15242 raft_consensus.cc:2229] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:47.276432 15242 raft_consensus.cc:2258] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:47.277469 15243 ts_tablet_manager.cc:1905] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:47.278519 15244 ts_tablet_manager.cc:1905] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:47.278580 15242 ts_tablet_manager.cc:1905] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 11:34:47 dist-test-slave-wl2h krb5kdc[14606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.65@KRBTEST.COM
I20260807 11:34:47.286293 15242 ts_tablet_manager.cc:1918] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:47.286582 15242 log.cc:1196] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/1f95910592184a25bf374f31a529feae
I20260807 11:34:47.287685 15242 ts_tablet_manager.cc:1939] T 1f95910592184a25bf374f31a529feae P bc77d02acdcf47feaba8032556a8edcc: Deleting consensus metadata
I20260807 11:34:47.289891 14637 catalog_manager.cc:4841] TS bc77d02acdcf47feaba8032556a8edcc (127.24.103.65:37531): tablet 1f95910592184a25bf374f31a529feae (table test-table [id=c453308f46144145bceca43712db8760]) successfully deleted
I20260807 11:34:47.290120 15243 ts_tablet_manager.cc:1918] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:47.290282 15243 log.cc:1196] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/1f95910592184a25bf374f31a529feae
I20260807 11:34:47.291729 15243 ts_tablet_manager.cc:1939] T 1f95910592184a25bf374f31a529feae P 31327cf436384f9293bef79a3d5a90b2: Deleting consensus metadata
I20260807 11:34:47.293238 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:47.262053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.24.103.65:37531)
0807 11:34:47.262396 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:47.262402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:47.272412 (+ 10010us) server_negotiation.cc:407] Connection header received
0807 11:34:47.272511 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:47.272517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:47.272624 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:47.272744 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:47.275529 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.276797 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.278839 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.279481 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.279598 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:47.285265 (+ 5667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:47.285281 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:47.285300 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:47.285342 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:47.289479 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:47.290534 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:47.290547 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:47.290549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:47.290687 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:47.291135 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:47.291140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:47.291143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:47.291645 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:47.291962 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:47.292603 (+ 641us) server_negotiation.cc:299] Negotiation successful
0807 11:34:47.292835 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 11:34:47.293756 14637 catalog_manager.cc:4841] TS 31327cf436384f9293bef79a3d5a90b2 (127.24.103.67:35423): tablet 1f95910592184a25bf374f31a529feae (table test-table [id=c453308f46144145bceca43712db8760]) successfully deleted
I20260807 11:34:47.294242 15244 ts_tablet_manager.cc:1918] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:47.294740 14877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:47.294423 15244 log.cc:1196] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/1f95910592184a25bf374f31a529feae
I20260807 11:34:47.296037 15244 ts_tablet_manager.cc:1939] T 1f95910592184a25bf374f31a529feae P d917e277b2254658ae274dab901cf410: Deleting consensus metadata
I20260807 11:34:47.298252 14642 catalog_manager.cc:4841] TS d917e277b2254658ae274dab901cf410 (127.24.103.66:39045): tablet 1f95910592184a25bf374f31a529feae (table test-table [id=c453308f46144145bceca43712db8760]) successfully deleted
I20260807 11:34:47.408250 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 14734
I20260807 11:34:47.430691 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 14890
Aug 07 11:34:47 dist-test-slave-wl2h krb5kdc[14606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.66@KRBTEST.COM
I20260807 11:34:47.449997 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:47.431878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.24.103.66:39045)
0807 11:34:47.432113 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:47.432120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:47.432145 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:34:47.432224 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:47.432229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:47.432314 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:47.432437 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:47.433935 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.435275 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.436908 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.437594 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.437729 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:47.442200 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:47.442215 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:47.442235 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:47.442291 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:47.446375 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:47.447484 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:47.447495 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:47.447499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:47.447708 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:47.448169 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:47.448175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:47.448179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:47.448663 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:47.449037 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:47.449357 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 11:34:47.449599 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:34:47.450959 15027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:47.569875 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 14890
I20260807 11:34:47.591791 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 15041
Aug 07 11:34:47 dist-test-slave-wl2h krb5kdc[14606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.67@KRBTEST.COM
I20260807 11:34:47.608980 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:47.592603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.24.103.67:35423)
0807 11:34:47.592798 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:47.592803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:47.592825 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:34:47.592996 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:47.593001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:47.593081 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:47.593174 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:47.594580 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.595503 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.596892 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:47.597341 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:47.597455 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:47.602563 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:47.602582 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:47.602600 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:47.602655 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:47.606076 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:47.606870 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:47.606878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:47.606881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:47.607117 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:47.607531 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:47.607534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:47.607537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:47.607916 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:47.608175 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:47.608484 (+ 309us) server_negotiation.cc:299] Negotiation successful
0807 11:34:47.608662 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 11:34:47.609687 15191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:47.701968 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 15041
I20260807 11:34:47.724387 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 14619
I20260807 11:34:47.725864 14694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:47.840979 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 14619
2026-08-07T11:34:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4987 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T11:34:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:34:47Z Disabled control of system clock
I20260807 11:34:47.942976 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:40065
--webserver_interface=127.24.103.126
--webserver_port=0
--builtin_ntp_servers=127.24.103.84:37355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:40065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:34:48.300904 15257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:48.301262 15257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:48.301365 15257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:48.309960 15257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:34:48.310097 15257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:48.310140 15257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:48.310221 15257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:34:48.310272 15257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:34:48.321321 15257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:40065
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:40065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:48.322762 15257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:48.324761 15257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:48.337806 15267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:48.337761 15265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:48.337780 15264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:48.339434 15257 server_base.cc:1034] running on GCE node
I20260807 11:34:48.340303 15257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:48.341881 15257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:48.343145 15257 hybrid_clock.cc:648] HybridClock initialized: now 1786102488343096 us; error 55 us; skew 500 ppm
I20260807 11:34:48.343554 15257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:48.345934 15257 webserver.cc:460] Webserver started at http://127.24.103.126:42269/ using document root <none> and password file <none>
I20260807 11:34:48.346660 15257 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:48.346794 15257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:48.347193 15257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:48.350212 15257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "85a85e6ecda44ee7830dd3355c3bcaea"
format_stamp: "Formatted at 2026-08-07 11:34:48 on dist-test-slave-wl2h"
server_key: "c1e759a82c8015dcffa1668c3f8af653"
server_key_iv: "10544c56b1d4cd8df677184916dedc41"
server_key_version: "encryptionkey@0"
I20260807 11:34:48.351161 15257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "85a85e6ecda44ee7830dd3355c3bcaea"
format_stamp: "Formatted at 2026-08-07 11:34:48 on dist-test-slave-wl2h"
server_key: "c1e759a82c8015dcffa1668c3f8af653"
server_key_iv: "10544c56b1d4cd8df677184916dedc41"
server_key_version: "encryptionkey@0"
I20260807 11:34:48.356472 15257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 11:34:48.360108 15273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:48.361526 15257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 11:34:48.361756 15257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "85a85e6ecda44ee7830dd3355c3bcaea"
format_stamp: "Formatted at 2026-08-07 11:34:48 on dist-test-slave-wl2h"
server_key: "c1e759a82c8015dcffa1668c3f8af653"
server_key_iv: "10544c56b1d4cd8df677184916dedc41"
server_key_version: "encryptionkey@0"
I20260807 11:34:48.362010 15257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:48.379745 15257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:48.380622 15257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:48.380923 15257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:48.400209 15257 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:40065
I20260807 11:34:48.400233 15333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:40065 every 8 connection(s)
I20260807 11:34:48.401618 15257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:34:48.405447 15334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:48.406612 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 15257
I20260807 11:34:48.406816 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:34:48.407428 24989 external_mini_cluster.cc:1428] Setting key ebcd738206aa3ff6d58b4ca615a0dc79
I20260807 11:34:48.419241 15334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea: Bootstrap starting.
I20260807 11:34:48.421461 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:48.409513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.24.103.126:40065)
0807 11:34:48.410719 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:48.410737 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:48.410782 (+ 45us) server_negotiation.cc:407] Connection header received
0807 11:34:48.412641 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:48.412663 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:48.413138 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:48.413549 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:48.414113 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:48.415454 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:48.416824 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:48.417507 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:48.417716 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:48.419004 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:48.419053 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:48.419076 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:48.419701 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:48.419787 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:48.419813 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:48.419941 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:48.420167 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 11:34:48.420496 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":449,"threads_started":1}
I20260807 11:34:48.422668 15334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:48.423837 15334 log.cc:824] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:48.426934 15334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea: No bootstrap required, opened a new log
I20260807 11:34:48.434852 15334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a85e6ecda44ee7830dd3355c3bcaea" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40065 } }
I20260807 11:34:48.435415 15334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:48.435535 15334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a85e6ecda44ee7830dd3355c3bcaea, State: Initialized, Role: FOLLOWER
I20260807 11:34:48.436196 15334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a85e6ecda44ee7830dd3355c3bcaea" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40065 } }
I20260807 11:34:48.436455 15334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:34:48.436610 15334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:34:48.436801 15334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:48.439693 15334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a85e6ecda44ee7830dd3355c3bcaea" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40065 } }
I20260807 11:34:48.440243 15334 leader_election.cc:302] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85a85e6ecda44ee7830dd3355c3bcaea; no voters:
I20260807 11:34:48.440856 15334 leader_election.cc:288] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:34:48.441076 15340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:48.442005 15340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [term 1 LEADER]: Becoming Leader. State: Replica: 85a85e6ecda44ee7830dd3355c3bcaea, State: Running, Role: LEADER
I20260807 11:34:48.442516 15340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a85e6ecda44ee7830dd3355c3bcaea" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40065 } }
I20260807 11:34:48.443248 15334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:34:48.445207 15342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a85e6ecda44ee7830dd3355c3bcaea. Latest consensus state: current_term: 1 leader_uuid: "85a85e6ecda44ee7830dd3355c3bcaea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a85e6ecda44ee7830dd3355c3bcaea" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40065 } } }
I20260807 11:34:48.445670 15342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:48.452242 15341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85a85e6ecda44ee7830dd3355c3bcaea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a85e6ecda44ee7830dd3355c3bcaea" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 40065 } } }
I20260807 11:34:48.452913 15341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea [sys.catalog]: This master's current role is: LEADER
W20260807 11:34:48.460783 15361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:34:48.460924 15361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:34:48.461088 15347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:34:48.462808 15347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:34:48.471071 15347 catalog_manager.cc:1347] Generated new cluster ID: ee8e274269e14a0d9a658aae6802b76c
I20260807 11:34:48.471232 15347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:34:48.488536 15347 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:34:48.489849 15347 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:34:48.495227 15347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea: Generated new TSK 0
I20260807 11:34:48.496313 15347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:34:48.510175 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.65:0
--local_ip_for_outbound_sockets=127.24.103.65
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40065
--builtin_ntp_servers=127.24.103.84:37355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 11:34:48.872330 15364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:48.872609 15364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:48.872803 15364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:48.880545 15364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:48.880662 15364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:48.880833 15364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.65
I20260807 11:34:48.890646 15364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.103.65
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.65
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:48.892077 15364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:48.894037 15364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:48.908993 15372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:48.909212 15369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:48.909247 15370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:48.909839 15364 server_base.cc:1034] running on GCE node
I20260807 11:34:48.910478 15364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:48.911629 15364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:48.912911 15364 hybrid_clock.cc:648] HybridClock initialized: now 1786102488912856 us; error 45 us; skew 500 ppm
I20260807 11:34:48.913273 15364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:48.915761 15364 webserver.cc:460] Webserver started at http://127.24.103.65:44069/ using document root <none> and password file <none>
I20260807 11:34:48.916469 15364 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:48.916608 15364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:48.917045 15364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:48.920087 15364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/data/instance:
uuid: "58d8b93d97de4caea78012d1f1ed9538"
format_stamp: "Formatted at 2026-08-07 11:34:48 on dist-test-slave-wl2h"
server_key: "7e2e14d682f4894add366307925981e1"
server_key_iv: "2c3f4db6d081fb350ce61d99e375dd1a"
server_key_version: "encryptionkey@0"
I20260807 11:34:48.920926 15364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance:
uuid: "58d8b93d97de4caea78012d1f1ed9538"
format_stamp: "Formatted at 2026-08-07 11:34:48 on dist-test-slave-wl2h"
server_key: "7e2e14d682f4894add366307925981e1"
server_key_iv: "2c3f4db6d081fb350ce61d99e375dd1a"
server_key_version: "encryptionkey@0"
I20260807 11:34:48.926347 15364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:34:48.930259 15378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:48.931869 15364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 11:34:48.932085 15364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/wal
uuid: "58d8b93d97de4caea78012d1f1ed9538"
format_stamp: "Formatted at 2026-08-07 11:34:48 on dist-test-slave-wl2h"
server_key: "7e2e14d682f4894add366307925981e1"
server_key_iv: "2c3f4db6d081fb350ce61d99e375dd1a"
server_key_version: "encryptionkey@0"
I20260807 11:34:48.932349 15364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:48.957692 15364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:48.958572 15364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:48.958899 15364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:48.959959 15364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:48.961465 15364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:48.961589 15364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:48.961719 15364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:48.961813 15364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:48.996490 15364 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.65:40557
I20260807 11:34:48.996579 15516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.65:40557 every 8 connection(s)
I20260807 11:34:48.997844 15364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/data/info.pb
I20260807 11:34:49.005997 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 15364
I20260807 11:34:49.006194 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/wal/instance
I20260807 11:34:49.006680 24989 external_mini_cluster.cc:1428] Setting key 54043efca8dea360f71c492db873abcb
I20260807 11:34:49.010033 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.66:0
--local_ip_for_outbound_sockets=127.24.103.66
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40065
--builtin_ntp_servers=127.24.103.84:37355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:34:49.011368 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:49.000445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:36825 (local address 127.24.103.126:40065)
0807 11:34:49.000918 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:49.000926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:49.002097 (+ 1171us) server_negotiation.cc:407] Connection header received
0807 11:34:49.003979 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:49.003983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:49.004053 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:49.004163 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:49.005260 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.006030 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.008055 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.008521 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.008657 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:49.010504 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:49.010509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:49.010513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:49.010531 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:49.010573 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:49.010584 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:49.010655 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:49.010928 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 11:34:49.011010 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
I20260807 11:34:49.012880 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:49.001032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40065 (local address 127.24.103.65:36825)
0807 11:34:49.001945 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:49.001981 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:49.003719 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:49.004367 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:49.004375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:49.004466 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:49.005074 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:49.005091 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.006233 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.006237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:49.007900 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:49.007912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.008855 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.008864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:49.009208 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:49.010093 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:34:49.010097 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:49.010173 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:34:49.010306 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:49.010687 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:49.010701 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:49.010712 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 11:34:49.010906 (+ 194us) client_negotiation.cc:240] Negotiation successful
0807 11:34:49.011397 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":279,"threads_started":1}
I20260807 11:34:49.014509 15517 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40065
I20260807 11:34:49.014824 15517 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:49.015642 15517 heartbeater.cc:511] Master 127.24.103.126:40065 requested a full tablet report, sending...
I20260807 11:34:49.017936 15290 ts_manager.cc:194] Registered new tserver with Master: 58d8b93d97de4caea78012d1f1ed9538 (127.24.103.65:40557)
I20260807 11:34:49.020138 15290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.65:36825
W20260807 11:34:49.352397 15522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:49.352710 15522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:49.352854 15522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:49.360226 15522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:49.360379 15522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:49.360569 15522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.66
I20260807 11:34:49.369910 15522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.103.66
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.66
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:49.371263 15522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:49.373129 15522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:49.390092 15528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:49.390223 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:49.390173 15527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:49.391325 15522 server_base.cc:1034] running on GCE node
I20260807 11:34:49.391968 15522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:49.393039 15522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:49.394276 15522 hybrid_clock.cc:648] HybridClock initialized: now 1786102489394236 us; error 34 us; skew 500 ppm
I20260807 11:34:49.394600 15522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:49.396898 15522 webserver.cc:460] Webserver started at http://127.24.103.66:39775/ using document root <none> and password file <none>
I20260807 11:34:49.397596 15522 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:49.397730 15522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:49.398051 15522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:49.400987 15522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/data/instance:
uuid: "3194d6a71c3c4547836963368e808bcf"
format_stamp: "Formatted at 2026-08-07 11:34:49 on dist-test-slave-wl2h"
server_key: "53d6f40f3071f23e92336ad854e55e4c"
server_key_iv: "398eb7e017df176cc2277b31044f44af"
server_key_version: "encryptionkey@0"
I20260807 11:34:49.401820 15522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance:
uuid: "3194d6a71c3c4547836963368e808bcf"
format_stamp: "Formatted at 2026-08-07 11:34:49 on dist-test-slave-wl2h"
server_key: "53d6f40f3071f23e92336ad854e55e4c"
server_key_iv: "398eb7e017df176cc2277b31044f44af"
server_key_version: "encryptionkey@0"
I20260807 11:34:49.407495 15522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 11:34:49.411177 15538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:49.412662 15522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:34:49.412850 15522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/wal
uuid: "3194d6a71c3c4547836963368e808bcf"
format_stamp: "Formatted at 2026-08-07 11:34:49 on dist-test-slave-wl2h"
server_key: "53d6f40f3071f23e92336ad854e55e4c"
server_key_iv: "398eb7e017df176cc2277b31044f44af"
server_key_version: "encryptionkey@0"
I20260807 11:34:49.413079 15522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:49.440299 15522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:49.441202 15522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:49.441522 15522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:49.442610 15522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:49.444371 15522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:49.444494 15522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:49.444597 15522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:49.444664 15522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:49.481678 15522 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.66:42071
I20260807 11:34:49.481791 15664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.66:42071 every 8 connection(s)
I20260807 11:34:49.483089 15522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/data/info.pb
I20260807 11:34:49.485911 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 15522
I20260807 11:34:49.486107 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/wal/instance
I20260807 11:34:49.486589 24989 external_mini_cluster.cc:1428] Setting key 79fcde251a5bd814b81940f27ecf7466
I20260807 11:34:49.489887 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.103.67:0
--local_ip_for_outbound_sockets=127.24.103.67
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40065
--builtin_ntp_servers=127.24.103.84:37355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 11:34:49.498368 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:49.485902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.66:51143 (local address 127.24.103.126:40065)
0807 11:34:49.486223 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:49.486230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:49.487634 (+ 1404us) server_negotiation.cc:407] Connection header received
0807 11:34:49.489699 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:49.489705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:49.489792 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:49.489912 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:49.491240 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.492449 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.494751 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.495506 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.495661 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:49.497334 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:49.497344 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:49.497351 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:49.497380 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:49.497425 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:49.497444 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:49.497543 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:49.497801 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 11:34:49.497925 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 11:34:49.499552 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:49.486401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40065 (local address 127.24.103.66:51143)
0807 11:34:49.487475 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:49.487520 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:49.489401 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:49.490261 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:49.490270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:49.490362 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:49.490969 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:49.490988 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.492757 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.492764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:49.494580 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:49.494599 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.495791 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.495801 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:49.496101 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:49.496899 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:34:49.496902 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:49.496956 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:34:49.497057 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:49.497560 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:49.497571 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:49.497580 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 11:34:49.497772 (+ 192us) client_negotiation.cc:240] Negotiation successful
0807 11:34:49.498200 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":341,"threads_started":1}
I20260807 11:34:49.501163 15665 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40065
I20260807 11:34:49.501503 15665 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:49.502426 15665 heartbeater.cc:511] Master 127.24.103.126:40065 requested a full tablet report, sending...
I20260807 11:34:49.504096 15290 ts_manager.cc:194] Registered new tserver with Master: 3194d6a71c3c4547836963368e808bcf (127.24.103.66:42071)
I20260807 11:34:49.504956 15290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.66:51143
W20260807 11:34:49.855788 15671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:49.856154 15671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:49.856324 15671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:49.863786 15671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:49.863965 15671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:49.864145 15671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.103.67
I20260807 11:34:49.873783 15671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.103.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.103.67
--webserver_port=0
--tserver_master_addrs=127.24.103.126:40065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.103.67
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:49.875353 15671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:49.877439 15671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:49.893117 15676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:49.893014 15677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:49.893035 15679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:49.893959 15671 server_base.cc:1034] running on GCE node
I20260807 11:34:49.894639 15671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:49.895718 15671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:49.896978 15671 hybrid_clock.cc:648] HybridClock initialized: now 1786102489896937 us; error 60 us; skew 500 ppm
I20260807 11:34:49.897369 15671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 11:34:49.899957 15671 webserver.cc:460] Webserver started at http://127.24.103.67:44263/ using document root <none> and password file <none>
I20260807 11:34:49.900719 15671 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:49.900854 15671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:49.901175 15671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:49.904690 15671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/data/instance:
uuid: "5aca6bcac5f649a1a479829e193e2bf8"
format_stamp: "Formatted at 2026-08-07 11:34:49 on dist-test-slave-wl2h"
server_key: "1af3e44acdf5995f24fe7eae875d0b10"
server_key_iv: "86c760814e65f3bc23b8f1f06c7ab7ad"
server_key_version: "encryptionkey@0"
I20260807 11:34:49.905587 15671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance:
uuid: "5aca6bcac5f649a1a479829e193e2bf8"
format_stamp: "Formatted at 2026-08-07 11:34:49 on dist-test-slave-wl2h"
server_key: "1af3e44acdf5995f24fe7eae875d0b10"
server_key_iv: "86c760814e65f3bc23b8f1f06c7ab7ad"
server_key_version: "encryptionkey@0"
I20260807 11:34:49.911489 15671 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 11:34:49.915357 15686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:49.917008 15671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:34:49.917222 15671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/wal
uuid: "5aca6bcac5f649a1a479829e193e2bf8"
format_stamp: "Formatted at 2026-08-07 11:34:49 on dist-test-slave-wl2h"
server_key: "1af3e44acdf5995f24fe7eae875d0b10"
server_key_iv: "86c760814e65f3bc23b8f1f06c7ab7ad"
server_key_version: "encryptionkey@0"
I20260807 11:34:49.917479 15671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:49.936466 15671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:49.937508 15671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:49.937836 15671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:49.938887 15671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 11:34:49.940478 15671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 11:34:49.940572 15671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:49.940726 15671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 11:34:49.940835 15671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:49.977412 15671 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.67:46063
I20260807 11:34:49.977478 15813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.67:46063 every 8 connection(s)
I20260807 11:34:49.978822 15671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/data/info.pb
I20260807 11:34:49.987457 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 15671
I20260807 11:34:49.987675 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/wal/instance
I20260807 11:34:49.988179 24989 external_mini_cluster.cc:1428] Setting key 30d9ce60e7dfb3750ed45484ad77213a
I20260807 11:34:49.994210 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:49.981733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.67:51813 (local address 127.24.103.126:40065)
0807 11:34:49.981921 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:49.981925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:49.983864 (+ 1939us) server_negotiation.cc:407] Connection header received
0807 11:34:49.985933 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:49.985940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:49.986055 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:49.986180 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:49.987684 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.988731 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.991104 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.991746 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.991854 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:49.993450 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:49.993454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:49.993458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:49.993476 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:49.993507 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:49.993518 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:49.993592 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:49.993836 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 11:34:49.993920 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 11:34:49.995342 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:49.982283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:40065 (local address 127.24.103.67:51813)
0807 11:34:49.983637 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:49.983705 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:49.985602 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:49.986442 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:49.986451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:49.986556 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:34:49.987405 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:49.987439 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.988946 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.988952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:49.990909 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:49.990922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:49.991996 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:49.992002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:49.992308 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:49.993121 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 11:34:49.993124 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:34:49.993169 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 11:34:49.993257 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:34:49.993631 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:34:49.993639 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:34:49.993646 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 11:34:49.993816 (+ 170us) client_negotiation.cc:240] Negotiation successful
0807 11:34:49.994177 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":339,"threads_started":1}
I20260807 11:34:49.997125 15814 heartbeater.cc:347] Connected to a master server at 127.24.103.126:40065
I20260807 11:34:49.997429 15814 heartbeater.cc:464] Registering TS with master...
I20260807 11:34:49.998205 15814 heartbeater.cc:511] Master 127.24.103.126:40065 requested a full tablet report, sending...
I20260807 11:34:49.999823 15290 ts_manager.cc:194] Registered new tserver with Master: 5aca6bcac5f649a1a479829e193e2bf8 (127.24.103.67:46063)
I20260807 11:34:50.001149 15290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.103.67:51813
I20260807 11:34:50.004354 24989 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 11:34:50.016048 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.009022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.24.103.126:40065)
0807 11:34:50.009238 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:50.009245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:50.009434 (+ 189us) server_negotiation.cc:407] Connection header received
0807 11:34:50.009725 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:50.009732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:50.009822 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:50.009913 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:50.010959 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.011813 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.013369 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.013924 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.014039 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.015198 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:50.015204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:50.015207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:50.015227 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:50.015262 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:50.015274 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:50.015355 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:50.015699 (+ 344us) server_negotiation.cc:299] Negotiation successful
0807 11:34:50.015801 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:34:50.023293 15517 heartbeater.cc:503] Master 127.24.103.126:40065 was elected leader, sending a full tablet report...
I20260807 11:34:50.020498 15290 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 11:34:50.028373 15290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 11:34:50.052771 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.040940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.65:40557 (local address 127.0.0.1:39018)
0807 11:34:50.041713 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:50.041756 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:50.041952 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:50.043252 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:50.043263 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:50.043298 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:50.043810 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:50.043827 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.047202 (+ 3375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.047208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:50.049878 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:50.049891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.051664 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.051674 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:50.051945 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.052003 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 11:34:50.052171 (+ 168us) client_negotiation.cc:240] Negotiation successful
0807 11:34:50.052372 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":855,"threads_started":1}
I20260807 11:34:50.052780 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.041605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.24.103.65:40557)
0807 11:34:50.042548 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:50.042556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:50.042585 (+ 29us) server_negotiation.cc:407] Connection header received
0807 11:34:50.042686 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:50.042696 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:50.042954 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:50.044056 (+ 1102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:50.044329 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.047007 (+ 2678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.050063 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.051416 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.051569 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.051900 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:50.052228 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 11:34:50.052403 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":1639,"threads_started":1}
I20260807 11:34:50.056217 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.041040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46063 (local address 127.0.0.1:50048)
0807 11:34:50.045080 (+ 4040us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:50.045105 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:50.045244 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:50.047803 (+ 2559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:50.047808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:50.047836 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:50.048462 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:50.048474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.050515 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.050521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:50.053017 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:50.053038 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.055305 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.055312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:50.055527 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.055562 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 11:34:50.055643 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 11:34:50.055786 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3876,"mutex_wait_us":7,"thread_start_us":5155,"threads_started":1}
I20260807 11:34:50.057462 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.041221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.24.103.67:46063)
0807 11:34:50.047129 (+ 5908us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:50.047139 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:50.047179 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:34:50.047334 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:50.047346 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:50.047572 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:50.047796 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:50.048664 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.050318 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.053294 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.055013 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.056494 (+ 1481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.056800 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:50.056878 (+ 78us) server_negotiation.cc:299] Negotiation successful
0807 11:34:50.057046 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5619,"thread_start_us":5402,"threads_started":1}
I20260807 11:34:50.057755 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.043087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.24.103.66:42071)
0807 11:34:50.047320 (+ 4233us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:50.047328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:50.050440 (+ 3112us) server_negotiation.cc:407] Connection header received
0807 11:34:50.050588 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:50.050597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:50.050866 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:50.051118 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:50.051883 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.053234 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.055574 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.056657 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.056822 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.057035 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:50.057235 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 11:34:50.057386 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4076,"thread_start_us":3944,"threads_started":1}
I20260807 11:34:50.058001 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.042568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:42071 (local address 127.0.0.1:40580)
0807 11:34:50.050293 (+ 7725us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:50.050312 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:50.050436 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:50.051106 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:50.051111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:50.051145 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:50.051647 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:50.051661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.053393 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.053396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:50.055392 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:50.055408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.056854 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.056860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:50.057031 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.057071 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 11:34:50.057174 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 11:34:50.057307 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7598,"thread_start_us":6292,"threads_started":1}
I20260807 11:34:50.058374 15437 tablet_service.cc:1467] Processing CreateTablet for tablet 080ab472c2304217b17381ace23514f4 (DEFAULT_TABLE table=test-table [id=b0ddde8616bd450290f048818032594a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:50.060122 15437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080ab472c2304217b17381ace23514f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:50.061028 15739 tablet_service.cc:1467] Processing CreateTablet for tablet 080ab472c2304217b17381ace23514f4 (DEFAULT_TABLE table=test-table [id=b0ddde8616bd450290f048818032594a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:50.061641 15593 tablet_service.cc:1467] Processing CreateTablet for tablet 080ab472c2304217b17381ace23514f4 (DEFAULT_TABLE table=test-table [id=b0ddde8616bd450290f048818032594a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 11:34:50.062198 15739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080ab472c2304217b17381ace23514f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:50.063045 15593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080ab472c2304217b17381ace23514f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:50.074738 15832 tablet_bootstrap.cc:492] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8: Bootstrap starting.
I20260807 11:34:50.077961 15832 tablet_bootstrap.cc:654] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:50.079167 15832 log.cc:824] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:50.079638 15834 tablet_bootstrap.cc:492] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf: Bootstrap starting.
I20260807 11:34:50.081658 15833 tablet_bootstrap.cc:492] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538: Bootstrap starting.
I20260807 11:34:50.084349 15834 tablet_bootstrap.cc:654] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:50.084920 15833 tablet_bootstrap.cc:654] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:50.085975 15833 log.cc:824] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:50.086001 15834 log.cc:824] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:50.089615 15832 tablet_bootstrap.cc:492] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8: No bootstrap required, opened a new log
I20260807 11:34:50.089622 15834 tablet_bootstrap.cc:492] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf: No bootstrap required, opened a new log
I20260807 11:34:50.089890 15834 ts_tablet_manager.cc:1397] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260807 11:34:50.089891 15833 tablet_bootstrap.cc:492] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538: No bootstrap required, opened a new log
I20260807 11:34:50.089910 15832 ts_tablet_manager.cc:1397] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260807 11:34:50.090159 15833 ts_tablet_manager.cc:1397] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 11:34:50.097658 15834 raft_consensus.cc:348] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aca6bcac5f649a1a479829e193e2bf8" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46063 } } peers { permanent_uuid: "58d8b93d97de4caea78012d1f1ed9538" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40557 } } peers { permanent_uuid: "3194d6a71c3c4547836963368e808bcf" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42071 } }
I20260807 11:34:50.098070 15834 raft_consensus.cc:374] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:50.098177 15834 raft_consensus.cc:729] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3194d6a71c3c4547836963368e808bcf, State: Initialized, Role: FOLLOWER
I20260807 11:34:50.098827 15834 consensus_queue.cc:260] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aca6bcac5f649a1a479829e193e2bf8" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46063 } } peers { permanent_uuid: "58d8b93d97de4caea78012d1f1ed9538" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40557 } } peers { permanent_uuid: "3194d6a71c3c4547836963368e808bcf" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42071 } }
I20260807 11:34:50.100050 15665 heartbeater.cc:503] Master 127.24.103.126:40065 was elected leader, sending a full tablet report...
I20260807 11:34:50.100786 15834 ts_tablet_manager.cc:1428] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260807 11:34:50.100911 15833 raft_consensus.cc:348] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aca6bcac5f649a1a479829e193e2bf8" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46063 } } peers { permanent_uuid: "58d8b93d97de4caea78012d1f1ed9538" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40557 } } peers { permanent_uuid: "3194d6a71c3c4547836963368e808bcf" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42071 } }
I20260807 11:34:50.100910 15832 raft_consensus.cc:348] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aca6bcac5f649a1a479829e193e2bf8" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46063 } } peers { permanent_uuid: "58d8b93d97de4caea78012d1f1ed9538" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40557 } } peers { permanent_uuid: "3194d6a71c3c4547836963368e808bcf" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42071 } }
I20260807 11:34:50.101397 15833 raft_consensus.cc:374] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:50.101397 15832 raft_consensus.cc:374] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:50.101496 15832 raft_consensus.cc:729] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aca6bcac5f649a1a479829e193e2bf8, State: Initialized, Role: FOLLOWER
I20260807 11:34:50.101492 15833 raft_consensus.cc:729] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d8b93d97de4caea78012d1f1ed9538, State: Initialized, Role: FOLLOWER
I20260807 11:34:50.102105 15833 consensus_queue.cc:260] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aca6bcac5f649a1a479829e193e2bf8" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46063 } } peers { permanent_uuid: "58d8b93d97de4caea78012d1f1ed9538" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40557 } } peers { permanent_uuid: "3194d6a71c3c4547836963368e808bcf" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42071 } }
I20260807 11:34:50.102103 15832 consensus_queue.cc:260] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aca6bcac5f649a1a479829e193e2bf8" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46063 } } peers { permanent_uuid: "58d8b93d97de4caea78012d1f1ed9538" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40557 } } peers { permanent_uuid: "3194d6a71c3c4547836963368e808bcf" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42071 } }
I20260807 11:34:50.103717 15814 heartbeater.cc:503] Master 127.24.103.126:40065 was elected leader, sending a full tablet report...
I20260807 11:34:50.104049 15833 ts_tablet_manager.cc:1428] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260807 11:34:50.104650 15832 ts_tablet_manager.cc:1428] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8: Time spent starting tablet: real 0.014s user 0.008s sys 0.008s
I20260807 11:34:50.109643 15840 raft_consensus.cc:482] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 11:34:50.109930 15840 raft_consensus.cc:504] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aca6bcac5f649a1a479829e193e2bf8" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46063 } } peers { permanent_uuid: "58d8b93d97de4caea78012d1f1ed9538" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40557 } } peers { permanent_uuid: "3194d6a71c3c4547836963368e808bcf" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42071 } }
I20260807 11:34:50.111742 15840 leader_election.cc:288] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aca6bcac5f649a1a479829e193e2bf8 (127.24.103.67:46063), 3194d6a71c3c4547836963368e808bcf (127.24.103.66:42071)
I20260807 11:34:50.118943 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.112030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:57827 (local address 127.24.103.67:46063)
0807 11:34:50.112317 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:50.112326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:50.113140 (+ 814us) server_negotiation.cc:407] Connection header received
0807 11:34:50.113315 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:50.113321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:50.113394 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:50.113485 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:50.114106 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.114885 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.117153 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.118091 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.118251 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.118321 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:50.118606 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 11:34:50.118721 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 11:34:50.118952 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.112203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.103.65:42341 (local address 127.24.103.66:42071)
0807 11:34:50.112428 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:50.112436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:50.113328 (+ 892us) server_negotiation.cc:407] Connection header received
0807 11:34:50.113506 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:50.113511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:50.113605 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:50.113708 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 11:34:50.114323 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.115099 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.117095 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.118023 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.118182 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.118260 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:50.118548 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 11:34:50.118681 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 11:34:50.119073 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.112179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.67:46063 (local address 127.24.103.65:57827)
0807 11:34:50.113034 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:50.113053 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:50.113179 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:50.113511 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:50.113516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:50.113540 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:50.113959 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:50.113973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.115030 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.115036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:50.116952 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:50.116965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.118308 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.118314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:50.118435 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.118457 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 11:34:50.118568 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 11:34:50.118693 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":94,"thread_start_us":479,"threads_started":1}
I20260807 11:34:50.119426 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.112213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.66:42071 (local address 127.24.103.65:42341)
0807 11:34:50.113199 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0807 11:34:50.113241 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 11:34:50.113369 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:34:50.113735 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:34:50.113739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:34:50.113759 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 11:34:50.114153 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:50.114169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.115294 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.115297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:50.116905 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:34:50.116921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.118212 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.118218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:34:50.118397 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.118421 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 11:34:50.118494 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 11:34:50.118628 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":35,"thread_start_us":525,"threads_started":1}
I20260807 11:34:50.120128 15764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080ab472c2304217b17381ace23514f4" candidate_uuid: "58d8b93d97de4caea78012d1f1ed9538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aca6bcac5f649a1a479829e193e2bf8" is_pre_election: true
I20260807 11:34:50.120469 15615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080ab472c2304217b17381ace23514f4" candidate_uuid: "58d8b93d97de4caea78012d1f1ed9538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3194d6a71c3c4547836963368e808bcf" is_pre_election: true
I20260807 11:34:50.120630 15764 raft_consensus.cc:2454] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d8b93d97de4caea78012d1f1ed9538 in term 0.
I20260807 11:34:50.120935 15615 raft_consensus.cc:2454] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d8b93d97de4caea78012d1f1ed9538 in term 0.
I20260807 11:34:50.121306 15379 leader_election.cc:302] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d8b93d97de4caea78012d1f1ed9538, 5aca6bcac5f649a1a479829e193e2bf8; no voters:
I20260807 11:34:50.121758 15840 raft_consensus.cc:2781] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 11:34:50.121917 15840 raft_consensus.cc:482] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 11:34:50.122025 15840 raft_consensus.cc:3037] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:50.124471 15840 raft_consensus.cc:504] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aca6bcac5f649a1a479829e193e2bf8" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46063 } } peers { permanent_uuid: "58d8b93d97de4caea78012d1f1ed9538" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40557 } } peers { permanent_uuid: "3194d6a71c3c4547836963368e808bcf" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42071 } }
I20260807 11:34:50.125123 15840 leader_election.cc:288] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [CANDIDATE]: Term 1 election: Requested vote from peers 5aca6bcac5f649a1a479829e193e2bf8 (127.24.103.67:46063), 3194d6a71c3c4547836963368e808bcf (127.24.103.66:42071)
I20260807 11:34:50.125619 15764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080ab472c2304217b17381ace23514f4" candidate_uuid: "58d8b93d97de4caea78012d1f1ed9538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aca6bcac5f649a1a479829e193e2bf8"
I20260807 11:34:50.125626 15615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080ab472c2304217b17381ace23514f4" candidate_uuid: "58d8b93d97de4caea78012d1f1ed9538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3194d6a71c3c4547836963368e808bcf"
I20260807 11:34:50.125826 15615 raft_consensus.cc:3037] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:50.125819 15764 raft_consensus.cc:3037] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:50.128149 15615 raft_consensus.cc:2454] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d8b93d97de4caea78012d1f1ed9538 in term 1.
I20260807 11:34:50.128247 15764 raft_consensus.cc:2454] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d8b93d97de4caea78012d1f1ed9538 in term 1.
I20260807 11:34:50.128621 15382 leader_election.cc:302] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3194d6a71c3c4547836963368e808bcf, 58d8b93d97de4caea78012d1f1ed9538; no voters:
I20260807 11:34:50.128955 15840 raft_consensus.cc:2781] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:50.129432 15840 raft_consensus.cc:686] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 1 LEADER]: Becoming Leader. State: Replica: 58d8b93d97de4caea78012d1f1ed9538, State: Running, Role: LEADER
I20260807 11:34:50.129922 15840 consensus_queue.cc:237] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aca6bcac5f649a1a479829e193e2bf8" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46063 } } peers { permanent_uuid: "58d8b93d97de4caea78012d1f1ed9538" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40557 } } peers { permanent_uuid: "3194d6a71c3c4547836963368e808bcf" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42071 } }
I20260807 11:34:50.134941 15290 catalog_manager.cc:5495] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d8b93d97de4caea78012d1f1ed9538 (127.24.103.65). New cstate: current_term: 1 leader_uuid: "58d8b93d97de4caea78012d1f1ed9538" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aca6bcac5f649a1a479829e193e2bf8" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d8b93d97de4caea78012d1f1ed9538" member_type: VOTER last_known_addr { host: "127.24.103.65" port: 40557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3194d6a71c3c4547836963368e808bcf" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42071 } health_report { overall_health: UNKNOWN } } }
I20260807 11:34:50.154505 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.148999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.24.103.65:40557)
0807 11:34:50.149205 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:50.149209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:50.149316 (+ 107us) server_negotiation.cc:407] Connection header received
0807 11:34:50.149500 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:50.149503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:50.149595 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:50.149722 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:34:50.150034 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.150753 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.152294 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.152802 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.152917 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.153371 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:34:50.153918 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:34:50.154088 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:50.154211 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 11:34:50.154283 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 11:34:50.162883 15615 raft_consensus.cc:1261] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf [term 1 FOLLOWER]: Refusing update from remote peer 58d8b93d97de4caea78012d1f1ed9538: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:50.162890 15764 raft_consensus.cc:1261] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8 [term 1 FOLLOWER]: Refusing update from remote peer 58d8b93d97de4caea78012d1f1ed9538: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:34:50.163889 15845 consensus_queue.cc:1035] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3194d6a71c3c4547836963368e808bcf" member_type: VOTER last_known_addr { host: "127.24.103.66" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:34:50.164264 15840 consensus_queue.cc:1035] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aca6bcac5f649a1a479829e193e2bf8" member_type: VOTER last_known_addr { host: "127.24.103.67" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:34:50.176970 15849 mvcc.cc:205] Tried to move back new op lower bound from 7315875799697424384 to 7315875799575986176. Current Snapshot: MvccSnapshot[applied={T|T < 7315875799697424384}]
I20260807 11:34:50.180961 15851 mvcc.cc:205] Tried to move back new op lower bound from 7315875799697424384 to 7315875799575986176. Current Snapshot: MvccSnapshot[applied={T|T < 7315875799697424384}]
I20260807 11:34:50.197342 15290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:50.197615 15290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 11:34:50.201467 15290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85a85e6ecda44ee7830dd3355c3bcaea: Sending DeleteTablet for 3 replicas of tablet 080ab472c2304217b17381ace23514f4
I20260807 11:34:50.202734 15739 tablet_service.cc:1514] Processing DeleteTablet for tablet 080ab472c2304217b17381ace23514f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:50 UTC) from {username='slave'} at 127.0.0.1:50048
I20260807 11:34:50.203081 15593 tablet_service.cc:1514] Processing DeleteTablet for tablet 080ab472c2304217b17381ace23514f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:50 UTC) from {username='slave'} at 127.0.0.1:40580
I20260807 11:34:50.203346 15437 tablet_service.cc:1514] Processing DeleteTablet for tablet 080ab472c2304217b17381ace23514f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 11:34:50 UTC) from {username='slave'} at 127.0.0.1:39018
I20260807 11:34:50.204713 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 15364
I20260807 11:34:50.214130 15861 tablet_replica.cc:330] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf: stopping tablet replica
I20260807 11:34:50.214876 15862 tablet_replica.cc:330] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538: stopping tablet replica
I20260807 11:34:50.215730 15862 raft_consensus.cc:2229] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 1 LEADER]: Raft consensus shutting down.
I20260807 11:34:50.216384 15861 raft_consensus.cc:2229] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:50.217594 15862 raft_consensus.cc:2258] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:50.218163 15861 raft_consensus.cc:2258] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:50.222312 15861 ts_tablet_manager.cc:1905] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:50.222759 15862 ts_tablet_manager.cc:1905] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:50.230736 15862 ts_tablet_manager.cc:1918] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:50.231040 15862 log.cc:1196] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-0/wal/wals/080ab472c2304217b17381ace23514f4
I20260807 11:34:50.231787 15862 ts_tablet_manager.cc:1939] T 080ab472c2304217b17381ace23514f4 P 58d8b93d97de4caea78012d1f1ed9538: Deleting consensus metadata
I20260807 11:34:50.233852 15276 catalog_manager.cc:4841] TS 58d8b93d97de4caea78012d1f1ed9538 (127.24.103.65:40557): tablet 080ab472c2304217b17381ace23514f4 (table test-table [id=b0ddde8616bd450290f048818032594a]) successfully deleted
I20260807 11:34:50.234047 15860 tablet_replica.cc:330] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8: stopping tablet replica
I20260807 11:34:50.234496 15860 raft_consensus.cc:2229] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 11:34:50.234922 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.206383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.24.103.65:40557)
0807 11:34:50.206649 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:50.206659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:50.220670 (+ 14011us) server_negotiation.cc:407] Connection header received
0807 11:34:50.221200 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:50.221205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:50.221320 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:50.221405 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:50.222101 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.223326 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.225216 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.225938 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.226030 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.232295 (+ 6265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:50.232305 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:50.232322 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:50.233178 (+ 856us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:50.233238 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:50.233262 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:50.233345 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:50.234429 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0807 11:34:50.234559 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:34:50.235031 15860 raft_consensus.cc:2258] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 11:34:50.235697 15512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:50.237397 15860 ts_tablet_manager.cc:1905] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 11:34:50.240854 15861 ts_tablet_manager.cc:1918] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:50.241199 15861 log.cc:1196] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-1/wal/wals/080ab472c2304217b17381ace23514f4
I20260807 11:34:50.241999 15861 ts_tablet_manager.cc:1939] T 080ab472c2304217b17381ace23514f4 P 3194d6a71c3c4547836963368e808bcf: Deleting consensus metadata
I20260807 11:34:50.244009 15860 ts_tablet_manager.cc:1918] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 11:34:50.244148 15276 catalog_manager.cc:4841] TS 3194d6a71c3c4547836963368e808bcf (127.24.103.66:42071): tablet 080ab472c2304217b17381ace23514f4 (table test-table [id=b0ddde8616bd450290f048818032594a]) successfully deleted
I20260807 11:34:50.244232 15860 log.cc:1196] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8: Deleting WAL directory at /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786102153172159-24989-0/minicluster-data/ts-2/wal/wals/080ab472c2304217b17381ace23514f4
I20260807 11:34:50.244684 15860 ts_tablet_manager.cc:1939] T 080ab472c2304217b17381ace23514f4 P 5aca6bcac5f649a1a479829e193e2bf8: Deleting consensus metadata
I20260807 11:34:50.246059 15274 catalog_manager.cc:4841] TS 5aca6bcac5f649a1a479829e193e2bf8 (127.24.103.67:46063): tablet 080ab472c2304217b17381ace23514f4 (table test-table [id=b0ddde8616bd450290f048818032594a]) successfully deleted
I20260807 11:34:50.340448 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 15364
I20260807 11:34:50.362332 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 15522
I20260807 11:34:50.369408 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.363253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40588 (local address 127.24.103.66:42071)
0807 11:34:50.363433 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:50.363438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:50.363458 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:34:50.363624 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:50.363628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:50.363733 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:50.363831 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:50.364188 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.365028 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.366469 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.366902 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.367057 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.368127 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:50.368137 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:50.368150 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:50.368734 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:50.368785 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:50.368805 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:50.368896 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:50.369025 (+ 129us) server_negotiation.cc:299] Negotiation successful
0807 11:34:50.369155 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:34:50.370174 15660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:50.474839 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 15522
I20260807 11:34:50.496383 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 15671
I20260807 11:34:50.503830 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:50.497202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.24.103.67:46063)
0807 11:34:50.497377 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:50.497381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:50.497401 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:34:50.497631 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:50.497648 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:50.497758 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:50.497862 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:50.498333 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.499168 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.500497 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:50.500976 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:50.501079 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:50.502338 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:50.502347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:50.502361 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:50.502940 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:50.502988 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:50.503222 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:50.503337 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:50.503480 (+ 143us) server_negotiation.cc:299] Negotiation successful
0807 11:34:50.503570 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 11:34:50.504494 15809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:50.594302 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 15671
I20260807 11:34:50.615481 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 15257
I20260807 11:34:50.616655 15326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:50.733966 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 15257
2026-08-07T11:34:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2892 ms)
[----------] 6 tests from EncryptionPolicyTest (23684 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:34:50 dist-test-slave-wl2h krb5kdc[15871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:34:50 dist-test-slave-wl2h krb5kdc[15871](info): set up 2 sockets
Aug 07 11:34:50 dist-test-slave-wl2h krb5kdc[15871](info): commencing operation
krb5kdc: starting...
W20260807 11:34:52.006157 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.002s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:34:52 dist-test-slave-wl2h krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:34:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:34:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:34:52.360438 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:52.727262 15886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:52.727716 15886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:52.727905 15886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:52.736438 15886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:34:52.736605 15886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:52.736720 15886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:52.736838 15886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:34:52.736927 15886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:34:52.748018 15886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:52.749650 15886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:52.751641 15886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:52.764835 15892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:52.764811 15895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:52.764870 15893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:52.766040 15886 server_base.cc:1034] running on GCE node
I20260807 11:34:52.766952 15886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:52.768687 15886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:52.769959 15886 hybrid_clock.cc:648] HybridClock initialized: now 1786102492769870 us; error 88 us; skew 500 ppm
I20260807 11:34:52.770356 15886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:52 dist-test-slave-wl2h krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:52.774137 15886 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:34:52.775962 15886 webserver.cc:460] Webserver started at http://127.0.0.1:42083/ using document root <none> and password file <none>
I20260807 11:34:52.776726 15886 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:52.776890 15886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:52.777226 15886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:52.780575 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "7a60632334254267af70f7e4a85c4780"
format_stamp: "Formatted at 2026-08-07 11:34:52 on dist-test-slave-wl2h"
server_key: "eb31d77af6f38d4f961c8e3374a423df"
server_key_iv: "3ec1c08efda707de740ee8dae20c0747"
server_key_version: "encryptionkey@0"
I20260807 11:34:52.781441 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "7a60632334254267af70f7e4a85c4780"
format_stamp: "Formatted at 2026-08-07 11:34:52 on dist-test-slave-wl2h"
server_key: "eb31d77af6f38d4f961c8e3374a423df"
server_key_iv: "3ec1c08efda707de740ee8dae20c0747"
server_key_version: "encryptionkey@0"
I20260807 11:34:52.788185 15886 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260807 11:34:52.792503 15904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:52.794390 15886 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260807 11:34:52.794633 15886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "7a60632334254267af70f7e4a85c4780"
format_stamp: "Formatted at 2026-08-07 11:34:52 on dist-test-slave-wl2h"
server_key: "eb31d77af6f38d4f961c8e3374a423df"
server_key_iv: "3ec1c08efda707de740ee8dae20c0747"
server_key_version: "encryptionkey@0"
I20260807 11:34:52.794899 15886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:52.840494 15886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:52.844976 15886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:52.845377 15886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:52.865944 15886 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43341
I20260807 11:34:52.865981 15963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43341 every 8 connection(s)
I20260807 11:34:52.867677 15886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:34:52.871843 15964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:52.876327 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 15886
I20260807 11:34:52.876657 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:34:52.877395 24989 external_mini_cluster.cc:1428] Setting key c11bfd50dcd9a765bc36a4195e8e09f5
I20260807 11:34:52.884825 15964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780: Bootstrap starting.
I20260807 11:34:52.888912 15964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:52.890195 15964 log.cc:824] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:52.893649 15964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780: No bootstrap required, opened a new log
Aug 07 11:34:52 dist-test-slave-wl2h krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 11:34:52.906098 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:52.880027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41539 (local address 127.0.0.1:43341)
0807 11:34:52.881013 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:52.881030 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:52.881082 (+ 52us) server_negotiation.cc:407] Connection header received
0807 11:34:52.883322 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:52.883356 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:52.883808 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:52.884157 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:52.886272 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:52.887782 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:52.889702 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:52.890714 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:52.890973 (+ 259us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:52.895521 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:52.895535 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:52.895563 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:52.895627 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:52.900568 (+ 4941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:52.901620 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:52.901637 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:52.901648 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:52.901887 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:52.902514 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:52.902520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:52.902524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:52.903391 (+ 867us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:52.903760 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:52.904250 (+ 490us) server_negotiation.cc:299] Negotiation successful
0807 11:34:52.904661 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":490,"threads_started":1}
I20260807 11:34:52.905764 15964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60632334254267af70f7e4a85c4780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260807 11:34:52.906313 15964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:52.906478 15964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a60632334254267af70f7e4a85c4780, State: Initialized, Role: FOLLOWER
I20260807 11:34:52.907181 15964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60632334254267af70f7e4a85c4780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260807 11:34:52.907450 15964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:34:52.907598 15964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:34:52.907841 15964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:52.911077 15964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60632334254267af70f7e4a85c4780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260807 11:34:52.911620 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:34:52.911636 15964 leader_election.cc:302] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a60632334254267af70f7e4a85c4780; no voters:
I20260807 11:34:52.912425 15964 leader_election.cc:288] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:34:52.912599 15970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:52.913247 15970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [term 1 LEADER]: Becoming Leader. State: Replica: 7a60632334254267af70f7e4a85c4780, State: Running, Role: LEADER
I20260807 11:34:52.913908 15970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60632334254267af70f7e4a85c4780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260807 11:34:52.920866 15972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a60632334254267af70f7e4a85c4780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60632334254267af70f7e4a85c4780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } } }
I20260807 11:34:52.921432 15972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:52.925863 15970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a60632334254267af70f7e4a85c4780. Latest consensus state: current_term: 1 leader_uuid: "7a60632334254267af70f7e4a85c4780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60632334254267af70f7e4a85c4780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } } }
I20260807 11:34:52.926637 15970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:52.928284 15964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:34:52.930289 15977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:34:52.937582 15977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:34:52.952329 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:52.932126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39995 (local address 127.0.0.1:43341)
0807 11:34:52.932398 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:52.932407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:52.933030 (+ 623us) server_negotiation.cc:407] Connection header received
0807 11:34:52.933354 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:52.933365 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:52.933466 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:52.933548 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:52.935869 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:52.937124 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:52.939163 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:52.939891 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:52.939994 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:52.943904 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:52.943911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:52.943916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:52.943961 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:52.948069 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:52.949260 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:52.949266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:52.949273 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:52.949434 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:52.950078 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:52.950083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:52.950086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:52.950616 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:52.950924 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:52.951739 (+ 815us) server_negotiation.cc:299] Negotiation successful
0807 11:34:52.951936 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260807 11:34:52.952579 15977 catalog_manager.cc:1347] Generated new cluster ID: eb8b4c02503a4b16a4f0d4723ff1dc25
I20260807 11:34:52.952811 15977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:34:52.964815 15977 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:34:52.966866 15977 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:34:52.984529 15977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a60632334254267af70f7e4a85c4780: Generated new TSK 0
I20260807 11:34:52.985662 15977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:34:52.999665 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 15886
I20260807 11:34:53.001114 15958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:53.097051 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 15886
W20260807 11:34:53.120512 15869 connection.cc:535] client connection to 127.0.0.1:43341 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43341): Connection reset by peer (error 104)
2026-08-07T11:34:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:34:53 dist-test-slave-wl2h krb5kdc[16000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:34:53 dist-test-slave-wl2h krb5kdc[16000](info): set up 2 sockets
Aug 07 11:34:53 dist-test-slave-wl2h krb5kdc[16000](info): commencing operation
krb5kdc: starting...
W20260807 11:34:54.356297 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:34:54 dist-test-slave-wl2h krb5kdc[16000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:34:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:34:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:34:54.703116 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:55.074591 16014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:55.074937 16014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:55.075138 16014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:55.083575 16014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:34:55.083732 16014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:55.083855 16014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:55.083992 16014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:34:55.084055 16014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:34:55.095815 16014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:55.097339 16014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:55.098829 16014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:55.111856 16019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:55.111902 16020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:55.111930 16024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:55.113547 16014 server_base.cc:1034] running on GCE node
I20260807 11:34:55.114395 16014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:55.115998 16014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:55.117272 16014 hybrid_clock.cc:648] HybridClock initialized: now 1786102495117221 us; error 39 us; skew 500 ppm
I20260807 11:34:55.117722 16014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:55 dist-test-slave-wl2h krb5kdc[16000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:55.127544 16014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:34:55.129302 16014 webserver.cc:460] Webserver started at http://127.0.0.1:44811/ using document root <none> and password file <none>
I20260807 11:34:55.130065 16014 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:55.130223 16014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:55.130575 16014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:55.134011 16014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "ca0e83a60d8745d7ac1441005faa5785"
format_stamp: "Formatted at 2026-08-07 11:34:55 on dist-test-slave-wl2h"
server_key: "b21023c5b3da8df96535313be736085b"
server_key_iv: "e8b09f68ce358ab4be2771da22c36446"
server_key_version: "encryptionkey@0"
I20260807 11:34:55.134863 16014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "ca0e83a60d8745d7ac1441005faa5785"
format_stamp: "Formatted at 2026-08-07 11:34:55 on dist-test-slave-wl2h"
server_key: "b21023c5b3da8df96535313be736085b"
server_key_iv: "e8b09f68ce358ab4be2771da22c36446"
server_key_version: "encryptionkey@0"
I20260807 11:34:55.140789 16014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 11:34:55.144665 16031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:55.146133 16014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 11:34:55.146327 16014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "ca0e83a60d8745d7ac1441005faa5785"
format_stamp: "Formatted at 2026-08-07 11:34:55 on dist-test-slave-wl2h"
server_key: "b21023c5b3da8df96535313be736085b"
server_key_iv: "e8b09f68ce358ab4be2771da22c36446"
server_key_version: "encryptionkey@0"
I20260807 11:34:55.146546 16014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:55.176091 16014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:55.180998 16014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:55.181340 16014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:55.201808 16014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40407
I20260807 11:34:55.201858 16095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40407 every 8 connection(s)
I20260807 11:34:55.203303 16014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:34:55.207440 16096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:55.209951 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 16014
I20260807 11:34:55.210166 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:34:55.210748 24989 external_mini_cluster.cc:1428] Setting key 983a09ef99f0a7d34f1f1b11cd1c2271
I20260807 11:34:55.220582 16096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785: Bootstrap starting.
I20260807 11:34:55.224474 16096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:55.225600 16096 log.cc:824] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:55.225679 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:55.212839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36821 (local address 127.0.0.1:40407)
0807 11:34:55.213748 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:55.213759 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:55.213801 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:34:55.215599 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:55.215621 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:55.216023 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:55.216356 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:55.216969 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:55.218393 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:55.220398 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:55.221223 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:55.221372 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:55.222945 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:55.222954 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:55.222968 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:55.223682 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:55.223769 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:55.223808 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:55.223952 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:55.224334 (+ 382us) server_negotiation.cc:299] Negotiation successful
0807 11:34:55.224575 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":451,"threads_started":1}
I20260807 11:34:55.228796 16096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785: No bootstrap required, opened a new log
I20260807 11:34:55.229964 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:34:55.239542 16096 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0e83a60d8745d7ac1441005faa5785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40407 } }
I20260807 11:34:55.240068 16096 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:55.240180 16096 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca0e83a60d8745d7ac1441005faa5785, State: Initialized, Role: FOLLOWER
I20260807 11:34:55.240854 16096 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0e83a60d8745d7ac1441005faa5785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40407 } }
I20260807 11:34:55.241153 16096 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:34:55.241335 16096 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:34:55.241541 16096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:55.243403 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:55.235771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44157 (local address 127.0.0.1:40407)
0807 11:34:55.236013 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:55.236026 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:55.236514 (+ 488us) server_negotiation.cc:407] Connection header received
0807 11:34:55.236714 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:55.236720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:55.236956 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:55.237075 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:55.237800 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:55.238668 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:55.240581 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:55.241076 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:55.241234 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:55.242529 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:55.242534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:55.242540 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:55.242558 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:55.242607 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:55.242628 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:55.242719 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:55.242941 (+ 222us) server_negotiation.cc:299] Negotiation successful
0807 11:34:55.243017 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260807 11:34:55.244756 16096 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0e83a60d8745d7ac1441005faa5785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40407 } }
I20260807 11:34:55.245303 16096 leader_election.cc:302] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca0e83a60d8745d7ac1441005faa5785; no voters:
I20260807 11:34:55.245900 16096 leader_election.cc:288] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:34:55.246088 16106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:55.246845 16106 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [term 1 LEADER]: Becoming Leader. State: Replica: ca0e83a60d8745d7ac1441005faa5785, State: Running, Role: LEADER
I20260807 11:34:55.247457 16106 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0e83a60d8745d7ac1441005faa5785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40407 } }
I20260807 11:34:55.248245 16096 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:34:55.250795 16107 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca0e83a60d8745d7ac1441005faa5785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0e83a60d8745d7ac1441005faa5785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40407 } } }
I20260807 11:34:55.251325 16107 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:55.250538 16108 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca0e83a60d8745d7ac1441005faa5785. Latest consensus state: current_term: 1 leader_uuid: "ca0e83a60d8745d7ac1441005faa5785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0e83a60d8745d7ac1441005faa5785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40407 } } }
I20260807 11:34:55.252312 16115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:34:55.252354 16108 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:55.259074 16115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:34:55.268492 16115 catalog_manager.cc:1347] Generated new cluster ID: 2bf46e58db674c05b06e81dfe202db40
I20260807 11:34:55.268628 16115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:34:55.321223 16115 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:34:55.323176 16115 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:34:55.332865 16115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca0e83a60d8745d7ac1441005faa5785: Generated new TSK 0
I20260807 11:34:55.333909 16115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:34:55.346658 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16014
W20260807 11:34:55.348902 16090 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36821
W20260807 11:34:55.349692 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16014: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:34:55.349903 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 16014
W20260807 11:34:55.371712 15998 connection.cc:535] client connection to 127.0.0.1:40407 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40407): Connection reset by peer (error 104)
2026-08-07T11:34:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:34:55 dist-test-slave-wl2h krb5kdc[16131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:34:55 dist-test-slave-wl2h krb5kdc[16131](info): set up 2 sockets
Aug 07 11:34:55 dist-test-slave-wl2h krb5kdc[16131](info): commencing operation
krb5kdc: starting...
W20260807 11:34:56.577760 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:34:56 dist-test-slave-wl2h krb5kdc[16131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:34:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:34:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:34:56.935595 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:57.297498 16144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:57.297868 16144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:57.298029 16144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:57.305794 16144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:34:57.305989 16144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:57.306097 16144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:57.306212 16144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:34:57.306299 16144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:34:57.316654 16144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:57.318022 16144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:57.319420 16144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:57.331629 16151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:57.331776 16150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:57.331753 16154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:57.332363 16144 server_base.cc:1034] running on GCE node
I20260807 11:34:57.333262 16144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:57.334736 16144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:57.335971 16144 hybrid_clock.cc:648] HybridClock initialized: now 1786102497335877 us; error 90 us; skew 500 ppm
I20260807 11:34:57.336325 16144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:57 dist-test-slave-wl2h krb5kdc[16131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:57.339741 16144 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:34:57.341300 16144 webserver.cc:460] Webserver started at http://127.0.0.1:35503/ using document root <none> and password file <none>
I20260807 11:34:57.341990 16144 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:57.342128 16144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:57.342458 16144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:57.345567 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "0d375ed1daf54ded85ed2da38d6a8aa5"
format_stamp: "Formatted at 2026-08-07 11:34:57 on dist-test-slave-wl2h"
server_key: "c6fb8026d3c713896bb923ade6e028e8"
server_key_iv: "831fad279375f195a5da5724388886cb"
server_key_version: "encryptionkey@0"
I20260807 11:34:57.346731 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "0d375ed1daf54ded85ed2da38d6a8aa5"
format_stamp: "Formatted at 2026-08-07 11:34:57 on dist-test-slave-wl2h"
server_key: "c6fb8026d3c713896bb923ade6e028e8"
server_key_iv: "831fad279375f195a5da5724388886cb"
server_key_version: "encryptionkey@0"
I20260807 11:34:57.352264 16144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260807 11:34:57.355850 16162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:57.357216 16144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:34:57.357427 16144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "0d375ed1daf54ded85ed2da38d6a8aa5"
format_stamp: "Formatted at 2026-08-07 11:34:57 on dist-test-slave-wl2h"
server_key: "c6fb8026d3c713896bb923ade6e028e8"
server_key_iv: "831fad279375f195a5da5724388886cb"
server_key_version: "encryptionkey@0"
I20260807 11:34:57.357633 16144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:57.363857 16144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:57.368227 16144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:57.368533 16144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:57.390514 16144 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44093
I20260807 11:34:57.390604 16219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44093 every 8 connection(s)
I20260807 11:34:57.392040 16144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:34:57.392467 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 16144
I20260807 11:34:57.392653 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:34:57.393276 24989 external_mini_cluster.cc:1428] Setting key ecd1aa0cf9ed39a341930987ccca02c2
I20260807 11:34:57.396925 16220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:57.402127 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:57.395161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47531 (local address 127.0.0.1:44093)
0807 11:34:57.395978 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:57.395993 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:57.396043 (+ 50us) server_negotiation.cc:407] Connection header received
0807 11:34:57.398379 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:57.398405 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:57.398794 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:57.399234 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:57.400248 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:57.400256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:57.400272 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:57.400288 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:57.400377 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:57.400408 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:57.400623 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:57.400765 (+ 142us) server_negotiation.cc:299] Negotiation successful
0807 11:34:57.400940 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":344,"threads_started":1}
I20260807 11:34:57.405695 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:34:57.410702 16220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5: Bootstrap starting.
I20260807 11:34:57.414408 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:57.411529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59287 (local address 127.0.0.1:44093)
0807 11:34:57.411702 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:57.411707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:57.412245 (+ 538us) server_negotiation.cc:407] Connection header received
0807 11:34:57.412406 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:57.412409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:57.412464 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:57.412554 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:57.413651 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:57.413655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:57.413658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:34:57.413673 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:57.413716 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:34:57.413731 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:57.413826 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:57.413977 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 11:34:57.414025 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 11:34:57.414865 16220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:57.416007 16220 log.cc:824] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:57.418999 16220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5: No bootstrap required, opened a new log
I20260807 11:34:57.427789 16220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d375ed1daf54ded85ed2da38d6a8aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } }
I20260807 11:34:57.428402 16220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:57.428531 16220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d375ed1daf54ded85ed2da38d6a8aa5, State: Initialized, Role: FOLLOWER
I20260807 11:34:57.429232 16220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d375ed1daf54ded85ed2da38d6a8aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } }
I20260807 11:34:57.429481 16220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:34:57.429600 16220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:34:57.429744 16220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:57.432312 16220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d375ed1daf54ded85ed2da38d6a8aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } }
I20260807 11:34:57.432902 16220 leader_election.cc:302] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d375ed1daf54ded85ed2da38d6a8aa5; no voters:
I20260807 11:34:57.433574 16220 leader_election.cc:288] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:34:57.433722 16231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:57.434629 16231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 0d375ed1daf54ded85ed2da38d6a8aa5, State: Running, Role: LEADER
I20260807 11:34:57.435289 16231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d375ed1daf54ded85ed2da38d6a8aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } }
I20260807 11:34:57.436184 16220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:34:57.438833 16232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d375ed1daf54ded85ed2da38d6a8aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d375ed1daf54ded85ed2da38d6a8aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } } }
I20260807 11:34:57.439332 16232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:57.439284 16234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d375ed1daf54ded85ed2da38d6a8aa5. Latest consensus state: current_term: 1 leader_uuid: "0d375ed1daf54ded85ed2da38d6a8aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d375ed1daf54ded85ed2da38d6a8aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } } }
I20260807 11:34:57.439579 16234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5 [sys.catalog]: This master's current role is: LEADER
W20260807 11:34:57.456127 16249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:34:57.456290 16249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:34:57.456497 16239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:34:57.458320 16239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:34:57.467355 16239 catalog_manager.cc:1347] Generated new cluster ID: 77d92130c5be41e5b50d5b1b4d41333c
I20260807 11:34:57.467485 16239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:34:57.486881 16239 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:34:57.487447 16239 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:34:57.497514 16239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d375ed1daf54ded85ed2da38d6a8aa5: Generated new TSK 0
I20260807 11:34:57.498445 16239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:34:57.514371 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16144
W20260807 11:34:57.515656 16213 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47531
W20260807 11:34:57.516521 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16144: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:34:57.516723 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 16144
2026-08-07T11:34:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2169 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:34:57 dist-test-slave-wl2h krb5kdc[16255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:34:57 dist-test-slave-wl2h krb5kdc[16255](info): set up 2 sockets
Aug 07 11:34:57 dist-test-slave-wl2h krb5kdc[16255](info): commencing operation
krb5kdc: starting...
W20260807 11:34:58.749680 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:34:58 dist-test-slave-wl2h krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:34:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:34:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:34:59.100833 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:34:59.463362 16269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:34:59.463734 16269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:34:59.463910 16269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:34:59.471670 16269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:34:59.471860 16269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:34:59.471976 16269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:34:59.472115 16269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:34:59.472194 16269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:34:59.482710 16269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:34:59.484651 16269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:34:59.486196 16269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:34:59.498539 16274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:59.499455 16275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:34:59.500597 16277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:34:59.501194 16269 server_base.cc:1034] running on GCE node
I20260807 11:34:59.502116 16269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:34:59.503777 16269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:34:59.505019 16269 hybrid_clock.cc:648] HybridClock initialized: now 1786102499504962 us; error 83 us; skew 500 ppm
I20260807 11:34:59.505398 16269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:34:59 dist-test-slave-wl2h krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:34:59.508675 16269 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:34:59.510245 16269 webserver.cc:460] Webserver started at http://127.0.0.1:39789/ using document root <none> and password file <none>
I20260807 11:34:59.510969 16269 fs_manager.cc:359] Metadata directory not provided
I20260807 11:34:59.511166 16269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:34:59.511485 16269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:34:59.514523 16269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "10120c5ae1ea41ee91c85e77e9708df5"
format_stamp: "Formatted at 2026-08-07 11:34:59 on dist-test-slave-wl2h"
server_key: "4566a2dd656eb52d2f341711aff62e08"
server_key_iv: "7b3e63adccf00f0cb42f359ee32af31e"
server_key_version: "encryptionkey@0"
I20260807 11:34:59.515364 16269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "10120c5ae1ea41ee91c85e77e9708df5"
format_stamp: "Formatted at 2026-08-07 11:34:59 on dist-test-slave-wl2h"
server_key: "4566a2dd656eb52d2f341711aff62e08"
server_key_iv: "7b3e63adccf00f0cb42f359ee32af31e"
server_key_version: "encryptionkey@0"
I20260807 11:34:59.520710 16269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 11:34:59.524384 16285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:34:59.525728 16269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 11:34:59.525933 16269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "10120c5ae1ea41ee91c85e77e9708df5"
format_stamp: "Formatted at 2026-08-07 11:34:59 on dist-test-slave-wl2h"
server_key: "4566a2dd656eb52d2f341711aff62e08"
server_key_iv: "7b3e63adccf00f0cb42f359ee32af31e"
server_key_version: "encryptionkey@0"
I20260807 11:34:59.526146 16269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:34:59.545385 16269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:34:59.550401 16269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:34:59.550774 16269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:34:59.571278 16269 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46585
I20260807 11:34:59.571318 16341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46585 every 8 connection(s)
I20260807 11:34:59.572784 16269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:34:59.576542 16342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:34:59.576871 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 16269
I20260807 11:34:59.577060 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:34:59.577524 24989 external_mini_cluster.cc:1428] Setting key 6f4c88f74f449f07051e3d3b85dc0422
I20260807 11:34:59.591292 16342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5: Bootstrap starting.
Aug 07 11:34:59 dist-test-slave-wl2h krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 11:34:59.595208 16342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5: Neither blocks nor log segments found. Creating new log.
I20260807 11:34:59.596678 16342 log.cc:824] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5: Log is configured to *not* fsync() on all Append() calls
I20260807 11:34:59.600282 16342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5: No bootstrap required, opened a new log
I20260807 11:34:59.603377 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:59.579350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50523 (local address 127.0.0.1:46585)
0807 11:34:59.580211 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:59.580222 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:59.580263 (+ 41us) server_negotiation.cc:407] Connection header received
0807 11:34:59.582571 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:59.582600 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:59.583066 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:59.583437 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:59.585618 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:59.587254 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:59.589153 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:59.589848 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:59.589989 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:59.595002 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:59.595040 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:59.595065 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:59.595119 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:59.599116 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:59.599817 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:59.599825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:59.599832 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:59.600045 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:59.600503 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:59.600507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:59.600509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:59.601066 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:59.601422 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:59.601866 (+ 444us) server_negotiation.cc:299] Negotiation successful
0807 11:34:59.602309 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":444,"threads_started":1}
I20260807 11:34:59.607088 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:34:59.608645 16342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10120c5ae1ea41ee91c85e77e9708df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260807 11:34:59.609021 16342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:34:59.609135 16342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10120c5ae1ea41ee91c85e77e9708df5, State: Initialized, Role: FOLLOWER
I20260807 11:34:59.609762 16342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10120c5ae1ea41ee91c85e77e9708df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260807 11:34:59.610052 16342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:34:59.610212 16342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:34:59.610387 16342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:34:59.613323 16342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10120c5ae1ea41ee91c85e77e9708df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260807 11:34:59.613822 16342 leader_election.cc:302] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10120c5ae1ea41ee91c85e77e9708df5; no voters:
I20260807 11:34:59.614470 16342 leader_election.cc:288] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:34:59.614581 16354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:34:59.616779 16342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:34:59.619329 16354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [term 1 LEADER]: Becoming Leader. State: Replica: 10120c5ae1ea41ee91c85e77e9708df5, State: Running, Role: LEADER
I20260807 11:34:59.620232 16354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10120c5ae1ea41ee91c85e77e9708df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260807 11:34:59.623282 16355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10120c5ae1ea41ee91c85e77e9708df5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10120c5ae1ea41ee91c85e77e9708df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } } }
I20260807 11:34:59.623654 16355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:59.622920 16357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10120c5ae1ea41ee91c85e77e9708df5. Latest consensus state: current_term: 1 leader_uuid: "10120c5ae1ea41ee91c85e77e9708df5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10120c5ae1ea41ee91c85e77e9708df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } } }
I20260807 11:34:59.624574 16357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5 [sys.catalog]: This master's current role is: LEADER
I20260807 11:34:59.633070 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:34:59.611725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59391 (local address 127.0.0.1:46585)
0807 11:34:59.611928 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 11:34:59.611935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:34:59.612361 (+ 426us) server_negotiation.cc:407] Connection header received
0807 11:34:59.612501 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:34:59.612508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:34:59.612584 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:34:59.612697 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:34:59.614308 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:59.615573 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:59.617384 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:34:59.618417 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:34:59.618678 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:34:59.622692 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:34:59.622702 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:34:59.622710 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:34:59.622763 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:34:59.627599 (+ 4836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:59.628760 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:59.628768 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:59.628775 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:59.628962 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:34:59.629579 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:34:59.629587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:34:59.629593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:34:59.630207 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:34:59.630512 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 11:34:59.632227 (+ 1715us) server_negotiation.cc:299] Negotiation successful
0807 11:34:59.632527 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 11:34:59.637676 16362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:34:59.643292 16362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:34:59.661736 16362 catalog_manager.cc:1347] Generated new cluster ID: 4b14cbaabf904a9fb00e11caabda5b1d
I20260807 11:34:59.661876 16362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:34:59.674746 16362 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:34:59.676712 16362 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:34:59.685580 16362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10120c5ae1ea41ee91c85e77e9708df5: Generated new TSK 0
I20260807 11:34:59.686754 16362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:34:59.702150 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16269
I20260807 11:34:59.703395 16337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:34:59.798542 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 16269
2026-08-07T11:34:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:34:59 dist-test-slave-wl2h krb5kdc[16380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:34:59 dist-test-slave-wl2h krb5kdc[16380](info): set up 2 sockets
Aug 07 11:34:59 dist-test-slave-wl2h krb5kdc[16380](info): commencing operation
krb5kdc: starting...
W20260807 11:35:01.038781 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:01 dist-test-slave-wl2h krb5kdc[16380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:35:01.384766 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:01.741148 16397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:01.741444 16397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:01.741626 16397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:01.748894 16397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:01.749011 16397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:01.749059 16397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:01.749208 16397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:01.749279 16397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:01.759869 16397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:01.761336 16397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:01.762725 16397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:01.776295 16403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:01.776322 16406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:01.776391 16404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:01.777729 16397 server_base.cc:1034] running on GCE node
I20260807 11:35:01.778594 16397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:01.780282 16397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:01.781589 16397 hybrid_clock.cc:648] HybridClock initialized: now 1786102501781528 us; error 47 us; skew 500 ppm
I20260807 11:35:01.781994 16397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:01 dist-test-slave-wl2h krb5kdc[16380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:01.792150 16397 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:35:01.793886 16397 webserver.cc:460] Webserver started at http://127.0.0.1:41913/ using document root <none> and password file <none>
I20260807 11:35:01.794629 16397 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:01.794800 16397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:01.795204 16397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:01.798513 16397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "53a3ef96d3c140ac9b97b3be35477fc4"
format_stamp: "Formatted at 2026-08-07 11:35:01 on dist-test-slave-wl2h"
server_key: "4174874a0d6dc1ec4d3267e7968788de"
server_key_iv: "b61a0218fefa6bee82438399eba6d22d"
server_key_version: "encryptionkey@0"
I20260807 11:35:01.799439 16397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "53a3ef96d3c140ac9b97b3be35477fc4"
format_stamp: "Formatted at 2026-08-07 11:35:01 on dist-test-slave-wl2h"
server_key: "4174874a0d6dc1ec4d3267e7968788de"
server_key_iv: "b61a0218fefa6bee82438399eba6d22d"
server_key_version: "encryptionkey@0"
I20260807 11:35:01.805424 16397 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:35:01.809310 16415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:01.810786 16397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:35:01.810998 16397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "53a3ef96d3c140ac9b97b3be35477fc4"
format_stamp: "Formatted at 2026-08-07 11:35:01 on dist-test-slave-wl2h"
server_key: "4174874a0d6dc1ec4d3267e7968788de"
server_key_iv: "b61a0218fefa6bee82438399eba6d22d"
server_key_version: "encryptionkey@0"
I20260807 11:35:01.811331 16397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:01.830720 16397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:01.835208 16397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:01.835520 16397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:01.855831 16397 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46567
I20260807 11:35:01.855885 16474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46567 every 8 connection(s)
I20260807 11:35:01.857326 16397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:01.859153 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 16397
I20260807 11:35:01.859359 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:01.859831 24989 external_mini_cluster.cc:1428] Setting key 6b5ead602747ebc667184dcdbcada2f4
I20260807 11:35:01.861917 16475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:01.875726 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:01.861849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58965 (local address 127.0.0.1:46567)
0807 11:35:01.862749 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:01.862765 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:01.862814 (+ 49us) server_negotiation.cc:407] Connection header received
0807 11:35:01.865063 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:01.865091 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:01.865637 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:01.866093 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:01.866606 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:01.868724 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:01.870141 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:01.871214 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:01.871395 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:01.872664 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:01.872674 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:01.872693 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:01.873478 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:01.873580 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:01.873623 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:01.873750 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:01.874001 (+ 251us) server_negotiation.cc:299] Negotiation successful
0807 11:35:01.874299 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":381,"threads_started":1}
I20260807 11:35:01.876986 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4: Bootstrap starting.
I20260807 11:35:01.880249 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:01.880934 16475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:01.882241 16475 log.cc:824] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:01.885588 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4: No bootstrap required, opened a new log
I20260807 11:35:01.892928 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:01.885766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60881 (local address 127.0.0.1:46567)
0807 11:35:01.886000 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:01.886007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:01.886411 (+ 404us) server_negotiation.cc:407] Connection header received
0807 11:35:01.886642 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:01.886645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:01.886705 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:01.886786 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:01.887465 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:01.888246 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:01.890212 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:01.890663 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:01.890763 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:01.892069 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:01.892075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:01.892078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:01.892101 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:01.892157 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:01.892179 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:01.892274 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:01.892464 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 11:35:01.892551 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 11:35:01.895654 16475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a3ef96d3c140ac9b97b3be35477fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } }
I20260807 11:35:01.896077 16475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:01.896196 16475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a3ef96d3c140ac9b97b3be35477fc4, State: Initialized, Role: FOLLOWER
I20260807 11:35:01.896766 16475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a3ef96d3c140ac9b97b3be35477fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } }
I20260807 11:35:01.897023 16475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:01.897189 16475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:01.897399 16475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:01.899775 16475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a3ef96d3c140ac9b97b3be35477fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } }
I20260807 11:35:01.900307 16475 leader_election.cc:302] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53a3ef96d3c140ac9b97b3be35477fc4; no voters:
I20260807 11:35:01.901023 16475 leader_election.cc:288] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:01.901207 16486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:01.902115 16486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [term 1 LEADER]: Becoming Leader. State: Replica: 53a3ef96d3c140ac9b97b3be35477fc4, State: Running, Role: LEADER
I20260807 11:35:01.902729 16486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a3ef96d3c140ac9b97b3be35477fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } }
I20260807 11:35:01.903712 16475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:01.906059 16488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53a3ef96d3c140ac9b97b3be35477fc4. Latest consensus state: current_term: 1 leader_uuid: "53a3ef96d3c140ac9b97b3be35477fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a3ef96d3c140ac9b97b3be35477fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } } }
I20260807 11:35:01.906972 16488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:01.906200 16487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53a3ef96d3c140ac9b97b3be35477fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a3ef96d3c140ac9b97b3be35477fc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } } }
I20260807 11:35:01.908169 16487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:01.911401 16491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:01.917238 16491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:01.928171 16491 catalog_manager.cc:1347] Generated new cluster ID: 96b61789e4c1496eb36ed0b5a9fdce3f
I20260807 11:35:01.928306 16491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:01.945315 16491 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:01.946731 16491 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:01.962260 16491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53a3ef96d3c140ac9b97b3be35477fc4: Generated new TSK 0
I20260807 11:35:01.963263 16491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:01.972716 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16397
W20260807 11:35:01.973907 16469 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58965
W20260807 11:35:01.974856 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16397: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:35:01.975107 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 16397
2026-08-07T11:35:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2176 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:02 dist-test-slave-wl2h krb5kdc[16510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:02 dist-test-slave-wl2h krb5kdc[16510](info): set up 2 sockets
Aug 07 11:35:02 dist-test-slave-wl2h krb5kdc[16510](info): commencing operation
krb5kdc: starting...
W20260807 11:35:03.222800 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:03 dist-test-slave-wl2h krb5kdc[16510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:35:03.595715 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:03.971175 16525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:03.971552 16525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:03.971740 16525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:03.979995 16525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:03.980156 16525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:03.980216 16525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:03.980353 16525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:03.980400 16525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:03.991752 16525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:03.993307 16525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:03.994932 16525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:04.009954 16535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:04.009949 16532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:04.010053 16533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:04.010979 16525 server_base.cc:1034] running on GCE node
I20260807 11:35:04.011932 16525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:04.013592 16525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:04.015157 16525 hybrid_clock.cc:648] HybridClock initialized: now 1786102504015080 us; error 58 us; skew 500 ppm
I20260807 11:35:04.015625 16525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:04 dist-test-slave-wl2h krb5kdc[16510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:04.024567 16525 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:35:04.026539 16525 webserver.cc:460] Webserver started at http://127.0.0.1:35807/ using document root <none> and password file <none>
I20260807 11:35:04.027420 16525 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:04.027588 16525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:04.027946 16525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:04.031425 16525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "6905f7547d7b435f8e79763c9ef84800"
format_stamp: "Formatted at 2026-08-07 11:35:04 on dist-test-slave-wl2h"
server_key: "8cec8c69c19b823d2883bf1b66829b2c"
server_key_iv: "4c7c238b17ab942a9415c28cae01388c"
server_key_version: "encryptionkey@0"
I20260807 11:35:04.032369 16525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "6905f7547d7b435f8e79763c9ef84800"
format_stamp: "Formatted at 2026-08-07 11:35:04 on dist-test-slave-wl2h"
server_key: "8cec8c69c19b823d2883bf1b66829b2c"
server_key_iv: "4c7c238b17ab942a9415c28cae01388c"
server_key_version: "encryptionkey@0"
I20260807 11:35:04.038074 16525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 11:35:04.041975 16542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:04.043555 16525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:35:04.043807 16525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "6905f7547d7b435f8e79763c9ef84800"
format_stamp: "Formatted at 2026-08-07 11:35:04 on dist-test-slave-wl2h"
server_key: "8cec8c69c19b823d2883bf1b66829b2c"
server_key_iv: "4c7c238b17ab942a9415c28cae01388c"
server_key_version: "encryptionkey@0"
I20260807 11:35:04.044076 16525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:04.050278 16525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:04.055260 16525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:04.055583 16525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:04.076846 16525 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42579
I20260807 11:35:04.076850 16603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42579 every 8 connection(s)
I20260807 11:35:04.078357 16525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:04.082595 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:04.084472 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 16525
I20260807 11:35:04.084676 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:04.085184 24989 external_mini_cluster.cc:1428] Setting key a6c6a643ebb1a81702a995314ca8b106
I20260807 11:35:04.093993 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:04.087471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56587 (local address 127.0.0.1:42579)
0807 11:35:04.088401 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:04.088413 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:04.088463 (+ 50us) server_negotiation.cc:407] Connection header received
0807 11:35:04.090264 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:04.090290 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:04.090690 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:04.091195 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:04.092304 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:04.092312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:04.092325 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:04.092340 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:04.092403 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:04.092423 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:04.092689 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:04.092834 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 11:35:04.093051 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":379,"threads_started":1}
I20260807 11:35:04.099092 16605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800: Bootstrap starting.
I20260807 11:35:04.099893 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:04.102527 16605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:04.104027 16605 log.cc:824] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:04.107519 16605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800: No bootstrap required, opened a new log
I20260807 11:35:04.107954 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:04.105149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42759 (local address 127.0.0.1:42579)
0807 11:35:04.105381 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:04.105389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:04.105668 (+ 279us) server_negotiation.cc:407] Connection header received
0807 11:35:04.105882 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:04.105886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:04.105939 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:04.106051 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:04.107201 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:04.107207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:04.107218 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:04.107237 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:04.107289 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:04.107307 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:04.107445 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:04.107603 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 11:35:04.107670 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 11:35:04.116998 16605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905f7547d7b435f8e79763c9ef84800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42579 } }
I20260807 11:35:04.117487 16605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:04.117633 16605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6905f7547d7b435f8e79763c9ef84800, State: Initialized, Role: FOLLOWER
I20260807 11:35:04.118316 16605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905f7547d7b435f8e79763c9ef84800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42579 } }
I20260807 11:35:04.118628 16605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:04.118790 16605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:04.118986 16605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:04.121902 16605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905f7547d7b435f8e79763c9ef84800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42579 } }
I20260807 11:35:04.122449 16605 leader_election.cc:302] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6905f7547d7b435f8e79763c9ef84800; no voters:
I20260807 11:35:04.123199 16605 leader_election.cc:288] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:04.123343 16616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:04.124095 16616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [term 1 LEADER]: Becoming Leader. State: Replica: 6905f7547d7b435f8e79763c9ef84800, State: Running, Role: LEADER
I20260807 11:35:04.124724 16616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905f7547d7b435f8e79763c9ef84800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42579 } }
I20260807 11:35:04.125644 16605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:04.128059 16618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6905f7547d7b435f8e79763c9ef84800. Latest consensus state: current_term: 1 leader_uuid: "6905f7547d7b435f8e79763c9ef84800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905f7547d7b435f8e79763c9ef84800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42579 } } }
I20260807 11:35:04.128618 16618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:04.128504 16617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6905f7547d7b435f8e79763c9ef84800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905f7547d7b435f8e79763c9ef84800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42579 } } }
I20260807 11:35:04.129009 16617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:04.135139 16626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:04.141827 16626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:04.149843 16635 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 11:35:04.156036 16626 catalog_manager.cc:1347] Generated new cluster ID: 2b646d5250964aaab574f3a22bd4d89b
I20260807 11:35:04.156180 16626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:04.184926 16626 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:04.185546 16626 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:04.195989 16626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6905f7547d7b435f8e79763c9ef84800: Generated new TSK 0
I20260807 11:35:04.196918 16626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:04.208185 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16525
W20260807 11:35:04.209466 16597 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56587
W20260807 11:35:04.210259 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16525: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:35:04.210486 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 16525
2026-08-07T11:35:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:04 dist-test-slave-wl2h krb5kdc[16639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:04 dist-test-slave-wl2h krb5kdc[16639](info): set up 2 sockets
Aug 07 11:35:04 dist-test-slave-wl2h krb5kdc[16639](info): commencing operation
krb5kdc: starting...
W20260807 11:35:05.465746 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.002s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:05 dist-test-slave-wl2h krb5kdc[16639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:35:05.840427 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:06.224792 16656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:06.225248 16656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:06.225435 16656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:06.233083 16656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:06.233227 16656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:06.233349 16656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:06.233469 16656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:06.233533 16656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:06.244966 16656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:06.246585 16656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:06.248003 16656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:06.261615 16661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:06.261749 16662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:06.261749 16664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:06.263216 16656 server_base.cc:1034] running on GCE node
I20260807 11:35:06.264115 16656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:06.265771 16656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:06.267129 16656 hybrid_clock.cc:648] HybridClock initialized: now 1786102506267077 us; error 97 us; skew 500 ppm
I20260807 11:35:06.267545 16656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:06 dist-test-slave-wl2h krb5kdc[16639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:06.271366 16656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:35:06.273180 16656 webserver.cc:460] Webserver started at http://127.0.0.1:36893/ using document root <none> and password file <none>
I20260807 11:35:06.273911 16656 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:06.274075 16656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:06.274410 16656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:06.277808 16656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "679cf904fdb64bd082803d6e645bc87b"
format_stamp: "Formatted at 2026-08-07 11:35:06 on dist-test-slave-wl2h"
server_key: "a1ac62855dd120af13485d3b1eba6fe9"
server_key_iv: "f5ff991ee5d020cd1c4d94655b108509"
server_key_version: "encryptionkey@0"
I20260807 11:35:06.278642 16656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "679cf904fdb64bd082803d6e645bc87b"
format_stamp: "Formatted at 2026-08-07 11:35:06 on dist-test-slave-wl2h"
server_key: "a1ac62855dd120af13485d3b1eba6fe9"
server_key_iv: "f5ff991ee5d020cd1c4d94655b108509"
server_key_version: "encryptionkey@0"
I20260807 11:35:06.284356 16656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 11:35:06.288367 16671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:06.289891 16656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:35:06.290113 16656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "679cf904fdb64bd082803d6e645bc87b"
format_stamp: "Formatted at 2026-08-07 11:35:06 on dist-test-slave-wl2h"
server_key: "a1ac62855dd120af13485d3b1eba6fe9"
server_key_iv: "f5ff991ee5d020cd1c4d94655b108509"
server_key_version: "encryptionkey@0"
I20260807 11:35:06.290345 16656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:06.326884 16656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:06.331966 16656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:06.332324 16656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:06.354331 16656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45917
I20260807 11:35:06.354372 16727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45917 every 8 connection(s)
I20260807 11:35:06.355944 16656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:06.358484 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 16656
I20260807 11:35:06.358716 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:06.359287 24989 external_mini_cluster.cc:1428] Setting key 8b8648af77fb0a8539627711349045c3
I20260807 11:35:06.360461 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 11:35:06 dist-test-slave-wl2h krb5kdc[16639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 11:35:06.376493 16728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b: Bootstrap starting.
I20260807 11:35:06.381348 16728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:06.382736 16728 log.cc:824] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:06.386672 16728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b: No bootstrap required, opened a new log
I20260807 11:35:06.387439 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:06.361241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:45581 (local address 127.0.0.1:45917)
0807 11:35:06.362127 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:06.362145 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:06.362215 (+ 70us) server_negotiation.cc:407] Connection header received
0807 11:35:06.364429 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:06.364459 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:06.365060 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:06.365549 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:06.367106 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:06.369453 (+ 2347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:06.370859 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:06.372089 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:06.372353 (+ 264us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:06.377203 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:06.377218 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:06.377240 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:06.377290 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:06.381942 (+ 4652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:06.382910 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:06.382921 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:06.382932 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:06.383216 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:06.383983 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:06.383987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:06.383989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:06.384773 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:06.385106 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:06.385667 (+ 561us) server_negotiation.cc:299] Negotiation successful
0807 11:35:06.386010 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":350,"threads_started":1}
I20260807 11:35:06.391588 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:06.396749 16728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679cf904fdb64bd082803d6e645bc87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260807 11:35:06.397225 16728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:06.397363 16728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 679cf904fdb64bd082803d6e645bc87b, State: Initialized, Role: FOLLOWER
I20260807 11:35:06.398083 16728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679cf904fdb64bd082803d6e645bc87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260807 11:35:06.398391 16728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:06.398564 16728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:06.398754 16728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:06.401832 16728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679cf904fdb64bd082803d6e645bc87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260807 11:35:06.402417 16728 leader_election.cc:302] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 679cf904fdb64bd082803d6e645bc87b; no voters:
I20260807 11:35:06.403249 16728 leader_election.cc:288] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:06.403388 16738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:06.406415 16738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [term 1 LEADER]: Becoming Leader. State: Replica: 679cf904fdb64bd082803d6e645bc87b, State: Running, Role: LEADER
I20260807 11:35:06.406526 16728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:06.407151 16738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679cf904fdb64bd082803d6e645bc87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260807 11:35:06.412611 16740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "679cf904fdb64bd082803d6e645bc87b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679cf904fdb64bd082803d6e645bc87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } } }
I20260807 11:35:06.413102 16740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:06.413717 16739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 679cf904fdb64bd082803d6e645bc87b. Latest consensus state: current_term: 1 leader_uuid: "679cf904fdb64bd082803d6e645bc87b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679cf904fdb64bd082803d6e645bc87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } } }
I20260807 11:35:06.414016 16739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:06.414814 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:06.397115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:52239 (local address 127.0.0.1:45917)
0807 11:35:06.397313 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:06.397323 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:06.397910 (+ 587us) server_negotiation.cc:407] Connection header received
0807 11:35:06.398090 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:06.398094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:06.398161 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:06.398247 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:06.400277 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:06.401069 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:06.402794 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:06.403459 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:06.403613 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:06.406236 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:06.406245 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:06.406252 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:06.406304 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:06.409725 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:06.410820 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:06.410828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:06.410833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:06.411059 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:06.411769 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:06.411777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:06.411782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:06.412225 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:06.412524 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:06.413988 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0807 11:35:06.414254 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:35:06.420573 16748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:06.425154 16748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:06.434335 16748 catalog_manager.cc:1347] Generated new cluster ID: 0cbaf0d809e243cfa5b08611e72ce209
I20260807 11:35:06.434571 16748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:06.457976 16748 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:06.459527 16748 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:06.467291 16748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 679cf904fdb64bd082803d6e645bc87b: Generated new TSK 0
I20260807 11:35:06.468433 16748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:06.482947 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16656
I20260807 11:35:06.484912 16723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:35:06.583827 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 16656
W20260807 11:35:06.606686 16637 connection.cc:535] client connection to 127.0.0.1:45917 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45917): Connection reset by peer (error 104)
2026-08-07T11:35:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:06 dist-test-slave-wl2h krb5kdc[16763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:06 dist-test-slave-wl2h krb5kdc[16763](info): set up 2 sockets
Aug 07 11:35:06 dist-test-slave-wl2h krb5kdc[16763](info): commencing operation
krb5kdc: starting...
W20260807 11:35:07.859212 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.003s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:08 dist-test-slave-wl2h krb5kdc[16763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:35:08.215260 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:08.572842 16776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:08.573177 16776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:08.573276 16776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:08.580853 16776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:08.580984 16776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:08.581043 16776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:08.581107 16776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:08.581153 16776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:08.591406 16776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:08.592896 16776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:08.594214 16776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:08.607049 16782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:08.607005 16783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:08.607132 16785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:08.608191 16776 server_base.cc:1034] running on GCE node
I20260807 11:35:08.609093 16776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:08.610865 16776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:08.612272 16776 hybrid_clock.cc:648] HybridClock initialized: now 1786102508612301 us; error 121 us; skew 500 ppm
I20260807 11:35:08.612864 16776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:08 dist-test-slave-wl2h krb5kdc[16763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:08.621984 16776 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:35:08.623942 16776 webserver.cc:460] Webserver started at http://127.0.0.1:44931/ using document root <none> and password file <none>
I20260807 11:35:08.624694 16776 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:08.624815 16776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:08.625183 16776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:08.628779 16776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "8568b18fc5ae46739d3cedb8c3b61570"
format_stamp: "Formatted at 2026-08-07 11:35:08 on dist-test-slave-wl2h"
server_key: "f57742433d16b76fec45160646817cd7"
server_key_iv: "9fd1e1432baf629d7844735048dd3b3c"
server_key_version: "encryptionkey@0"
I20260807 11:35:08.629990 16776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "8568b18fc5ae46739d3cedb8c3b61570"
format_stamp: "Formatted at 2026-08-07 11:35:08 on dist-test-slave-wl2h"
server_key: "f57742433d16b76fec45160646817cd7"
server_key_iv: "9fd1e1432baf629d7844735048dd3b3c"
server_key_version: "encryptionkey@0"
I20260807 11:35:08.635787 16776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 11:35:08.639546 16794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:08.641094 16776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 11:35:08.641345 16776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "8568b18fc5ae46739d3cedb8c3b61570"
format_stamp: "Formatted at 2026-08-07 11:35:08 on dist-test-slave-wl2h"
server_key: "f57742433d16b76fec45160646817cd7"
server_key_iv: "9fd1e1432baf629d7844735048dd3b3c"
server_key_version: "encryptionkey@0"
I20260807 11:35:08.641563 16776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:08.659569 16776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:08.664741 16776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:08.665103 16776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:08.687702 16776 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45075
I20260807 11:35:08.687808 16851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45075 every 8 connection(s)
I20260807 11:35:08.689222 16776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:08.691991 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 16776
I20260807 11:35:08.692224 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:08.692857 24989 external_mini_cluster.cc:1428] Setting key df5d6869173c9d45c66f3c2c6cab56fd
I20260807 11:35:08.694165 16853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:08.707660 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:08.694823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:39039 (local address 127.0.0.1:45075)
0807 11:35:08.695727 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:08.695739 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:08.695782 (+ 43us) server_negotiation.cc:407] Connection header received
0807 11:35:08.697397 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:08.697419 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:08.697845 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:08.698192 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:08.698870 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:08.700360 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:08.702299 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:08.703074 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:08.703238 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:08.704708 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:08.704717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:08.704733 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:08.705476 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:08.705574 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:08.705606 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:08.705745 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:08.706233 (+ 488us) server_negotiation.cc:299] Negotiation successful
0807 11:35:08.706475 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":454,"threads_started":1}
I20260807 11:35:08.708673 16853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570: Bootstrap starting.
I20260807 11:35:08.712239 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:08.712446 16853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:08.713478 16853 log.cc:824] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:08.717234 16853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570: No bootstrap required, opened a new log
I20260807 11:35:08.725673 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:08.717752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:47415 (local address 127.0.0.1:45075)
0807 11:35:08.717976 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:08.717984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:08.718610 (+ 626us) server_negotiation.cc:407] Connection header received
0807 11:35:08.718792 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:08.718799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:08.718897 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:08.719060 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:08.719718 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:08.721080 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:08.722429 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:08.723343 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:08.723495 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:08.724717 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:08.724726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:08.724731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:08.724757 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:08.724808 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:08.724832 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:08.724976 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:08.725181 (+ 205us) server_negotiation.cc:299] Negotiation successful
0807 11:35:08.725241 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:35:08.728565 16853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8568b18fc5ae46739d3cedb8c3b61570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260807 11:35:08.729001 16853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:08.729142 16853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8568b18fc5ae46739d3cedb8c3b61570, State: Initialized, Role: FOLLOWER
I20260807 11:35:08.729841 16853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8568b18fc5ae46739d3cedb8c3b61570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260807 11:35:08.730101 16853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:08.730283 16853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:08.730485 16853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:08.733384 16853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8568b18fc5ae46739d3cedb8c3b61570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260807 11:35:08.733955 16853 leader_election.cc:302] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8568b18fc5ae46739d3cedb8c3b61570; no voters:
I20260807 11:35:08.734622 16853 leader_election.cc:288] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:08.734772 16863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:08.735674 16863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [term 1 LEADER]: Becoming Leader. State: Replica: 8568b18fc5ae46739d3cedb8c3b61570, State: Running, Role: LEADER
I20260807 11:35:08.736285 16863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8568b18fc5ae46739d3cedb8c3b61570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260807 11:35:08.736914 16853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:08.740096 16866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8568b18fc5ae46739d3cedb8c3b61570. Latest consensus state: current_term: 1 leader_uuid: "8568b18fc5ae46739d3cedb8c3b61570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8568b18fc5ae46739d3cedb8c3b61570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } } }
I20260807 11:35:08.740557 16866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:08.740221 16865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8568b18fc5ae46739d3cedb8c3b61570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8568b18fc5ae46739d3cedb8c3b61570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } } }
I20260807 11:35:08.741715 16865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:08.744293 16872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:08.750778 16872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:08.760740 16872 catalog_manager.cc:1347] Generated new cluster ID: 7faf97051ef646c2bab77a18afb1df2b
I20260807 11:35:08.760874 16872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:08.771401 16872 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:08.773113 16872 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:08.785226 16872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8568b18fc5ae46739d3cedb8c3b61570: Generated new TSK 0
I20260807 11:35:08.786419 16872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:08.801010 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16776
W20260807 11:35:08.802639 16846 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:39039
W20260807 11:35:08.803512 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16776: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:35:08.803696 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 16776
W20260807 11:35:08.825762 16761 connection.cc:535] client connection to 127.0.0.1:45075 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45075): Connection reset by peer (error 104)
2026-08-07T11:35:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:08 dist-test-slave-wl2h krb5kdc[16886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:08 dist-test-slave-wl2h krb5kdc[16886](info): set up 2 sockets
Aug 07 11:35:08 dist-test-slave-wl2h krb5kdc[16886](info): commencing operation
krb5kdc: starting...
W20260807 11:35:10.040752 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.005s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:10 dist-test-slave-wl2h krb5kdc[16886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:35:10.398696 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:10.757216 16901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:10.757570 16901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:10.757741 16901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:10.765378 16901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:10.765580 16901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:10.765709 16901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:10.765816 16901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:10.765906 16901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:10.776329 16901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:10.777722 16901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:10.779165 16901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:10.797829 16907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:10.797832 16906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:10.798111 16909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:10.800202 16901 server_base.cc:1034] running on GCE node
I20260807 11:35:10.801152 16901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:10.803105 16901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:10.804354 16901 hybrid_clock.cc:648] HybridClock initialized: now 1786102510804318 us; error 47 us; skew 500 ppm
I20260807 11:35:10.804744 16901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:10 dist-test-slave-wl2h krb5kdc[16886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:10.808290 16901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:35:10.809935 16901 webserver.cc:460] Webserver started at http://127.0.0.1:36011/ using document root <none> and password file <none>
I20260807 11:35:10.810696 16901 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:10.810858 16901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:10.811256 16901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:10.814507 16901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "8586c973e7f444fabccef844059142ce"
format_stamp: "Formatted at 2026-08-07 11:35:10 on dist-test-slave-wl2h"
server_key: "56048e1bb1f4f17c96add5f3f453c19d"
server_key_iv: "ae9974dcb3194825c36f9d7aba7c2b3f"
server_key_version: "encryptionkey@0"
I20260807 11:35:10.815398 16901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "8586c973e7f444fabccef844059142ce"
format_stamp: "Formatted at 2026-08-07 11:35:10 on dist-test-slave-wl2h"
server_key: "56048e1bb1f4f17c96add5f3f453c19d"
server_key_iv: "ae9974dcb3194825c36f9d7aba7c2b3f"
server_key_version: "encryptionkey@0"
I20260807 11:35:10.820819 16901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260807 11:35:10.824743 16917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:10.826200 16901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260807 11:35:10.826407 16901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "8586c973e7f444fabccef844059142ce"
format_stamp: "Formatted at 2026-08-07 11:35:10 on dist-test-slave-wl2h"
server_key: "56048e1bb1f4f17c96add5f3f453c19d"
server_key_iv: "ae9974dcb3194825c36f9d7aba7c2b3f"
server_key_version: "encryptionkey@0"
I20260807 11:35:10.826625 16901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:10.832756 16901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:10.837033 16901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:10.837361 16901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:10.857524 16901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41385
I20260807 11:35:10.857570 16978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41385 every 8 connection(s)
I20260807 11:35:10.858963 16901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:10.862794 16979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:10.863265 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 16901
I20260807 11:35:10.863466 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:10.864063 24989 external_mini_cluster.cc:1428] Setting key 7c2ea4319bdedb56bc87ffd9de79ebb7
I20260807 11:35:10.873153 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:10.865999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:43733 (local address 127.0.0.1:41385)
0807 11:35:10.866895 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:10.866907 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:10.866951 (+ 44us) server_negotiation.cc:407] Connection header received
0807 11:35:10.869448 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:10.869475 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:10.869841 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:10.870287 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:10.871427 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:10.871434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:10.871447 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:10.871463 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:10.871521 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:10.871545 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:10.871810 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:10.871955 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 11:35:10.872178 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":395,"threads_started":1}
I20260807 11:35:10.876698 16979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce: Bootstrap starting.
I20260807 11:35:10.877214 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:10.879935 16979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:10.880970 16979 log.cc:824] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:10.884429 16979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce: No bootstrap required, opened a new log
I20260807 11:35:10.885066 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:10.881896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:58329 (local address 127.0.0.1:41385)
0807 11:35:10.882038 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:10.882043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:10.882449 (+ 406us) server_negotiation.cc:407] Connection header received
0807 11:35:10.883144 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:10.883150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:10.883226 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:10.883341 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:10.884272 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:10.884277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:10.884282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:10.884302 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:10.884346 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:10.884365 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:10.884515 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:10.884666 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 11:35:10.884737 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260807 11:35:10.892679 16979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586c973e7f444fabccef844059142ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } }
I20260807 11:35:10.893044 16979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:10.893119 16979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586c973e7f444fabccef844059142ce, State: Initialized, Role: FOLLOWER
I20260807 11:35:10.893649 16979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586c973e7f444fabccef844059142ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } }
I20260807 11:35:10.893847 16979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:10.893954 16979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:10.894075 16979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:10.896586 16979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586c973e7f444fabccef844059142ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } }
I20260807 11:35:10.897046 16979 leader_election.cc:302] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8586c973e7f444fabccef844059142ce; no voters:
I20260807 11:35:10.897572 16979 leader_election.cc:288] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:10.897737 16993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:10.898826 16993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [term 1 LEADER]: Becoming Leader. State: Replica: 8586c973e7f444fabccef844059142ce, State: Running, Role: LEADER
I20260807 11:35:10.899344 16993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586c973e7f444fabccef844059142ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } }
I20260807 11:35:10.899657 16979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:10.902361 16996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8586c973e7f444fabccef844059142ce. Latest consensus state: current_term: 1 leader_uuid: "8586c973e7f444fabccef844059142ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586c973e7f444fabccef844059142ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } } }
I20260807 11:35:10.902522 16994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8586c973e7f444fabccef844059142ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586c973e7f444fabccef844059142ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } } }
I20260807 11:35:10.902791 16994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:10.902799 16996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:10.908362 17003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:10.913918 17003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:10.922410 17003 catalog_manager.cc:1347] Generated new cluster ID: 2fe644fe0d8541e0ba8364291ea15c61
I20260807 11:35:10.922547 17003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:10.938651 17003 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:10.939204 17003 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:10.958786 17003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8586c973e7f444fabccef844059142ce: Generated new TSK 0
I20260807 11:35:10.959724 17003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:10.971527 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16901
W20260807 11:35:10.972656 16974 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:43733
W20260807 11:35:10.973346 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 16901: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:35:10.973518 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 16901
2026-08-07T11:35:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2166 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:11 dist-test-slave-wl2h krb5kdc[17017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:11 dist-test-slave-wl2h krb5kdc[17017](info): set up 2 sockets
Aug 07 11:35:11 dist-test-slave-wl2h krb5kdc[17017](info): commencing operation
krb5kdc: starting...
W20260807 11:35:12.211341 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.008s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:12 dist-test-slave-wl2h krb5kdc[17017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:35:12.566982 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:12.916498 17031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:12.916851 17031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:12.917047 17031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:12.925081 17031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:12.925244 17031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:12.925343 17031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:12.925457 17031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:12.925529 17031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:12.936807 17031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:12.938283 17031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:12.939766 17031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:12.952860 17036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:12.952827 17037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:12.953596 17039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:12.954079 17031 server_base.cc:1034] running on GCE node
I20260807 11:35:12.954982 17031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:12.956709 17031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:12.957966 17031 hybrid_clock.cc:648] HybridClock initialized: now 1786102512957888 us; error 69 us; skew 500 ppm
I20260807 11:35:12.958359 17031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:12 dist-test-slave-wl2h krb5kdc[17017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:12.961928 17031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:35:12.963711 17031 webserver.cc:460] Webserver started at http://127.0.0.1:36061/ using document root <none> and password file <none>
I20260807 11:35:12.964427 17031 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:12.964591 17031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:12.964926 17031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:12.968243 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "8bddcb8b018745c38c842419c476b9d8"
format_stamp: "Formatted at 2026-08-07 11:35:12 on dist-test-slave-wl2h"
server_key: "1c804d625db84b127ead301badf86401"
server_key_iv: "a289d5e754cabe6afac36676df412741"
server_key_version: "encryptionkey@0"
I20260807 11:35:12.969076 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "8bddcb8b018745c38c842419c476b9d8"
format_stamp: "Formatted at 2026-08-07 11:35:12 on dist-test-slave-wl2h"
server_key: "1c804d625db84b127ead301badf86401"
server_key_iv: "a289d5e754cabe6afac36676df412741"
server_key_version: "encryptionkey@0"
I20260807 11:35:12.974511 17031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:35:12.978457 17047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:12.979997 17031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:35:12.980206 17031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "8bddcb8b018745c38c842419c476b9d8"
format_stamp: "Formatted at 2026-08-07 11:35:12 on dist-test-slave-wl2h"
server_key: "1c804d625db84b127ead301badf86401"
server_key_iv: "a289d5e754cabe6afac36676df412741"
server_key_version: "encryptionkey@0"
I20260807 11:35:12.980432 17031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:13.020552 17031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:13.025611 17031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:13.025949 17031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:13.046401 17031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36163
I20260807 11:35:13.046499 17104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36163 every 8 connection(s)
I20260807 11:35:13.047935 17031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:13.052089 17105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:13.053213 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 17031
I20260807 11:35:13.053438 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:13.054075 24989 external_mini_cluster.cc:1428] Setting key 36aa67487792613854871a3187d24e2b
I20260807 11:35:13.068624 17105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8: Bootstrap starting.
Aug 07 11:35:13 dist-test-slave-wl2h krb5kdc[17017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 11:35:13.072930 17105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:13.074232 17105 log.cc:824] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:13.077576 17105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8: No bootstrap required, opened a new log
I20260807 11:35:13.081486 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:13.056138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:48239 (local address 127.0.0.1:36163)
0807 11:35:13.056974 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:13.056985 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:13.057027 (+ 42us) server_negotiation.cc:407] Connection header received
0807 11:35:13.059155 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:13.059181 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:13.059588 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:13.059927 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:13.062106 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:13.063675 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:13.065811 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:13.066575 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:13.066724 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:13.072379 (+ 5655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:13.072388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:13.072403 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:13.072446 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:13.076215 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:13.077141 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:13.077148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:13.077154 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:13.077319 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:13.077851 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:13.077857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:13.077860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:13.078650 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:13.078985 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:13.079582 (+ 597us) server_negotiation.cc:299] Negotiation successful
0807 11:35:13.079978 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":384,"threads_started":1}
I20260807 11:35:13.085868 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:13.087993 17105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bddcb8b018745c38c842419c476b9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260807 11:35:13.088474 17105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:13.088608 17105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bddcb8b018745c38c842419c476b9d8, State: Initialized, Role: FOLLOWER
I20260807 11:35:13.089318 17105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bddcb8b018745c38c842419c476b9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260807 11:35:13.089670 17105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:13.089850 17105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:13.090040 17105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:13.093171 17105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bddcb8b018745c38c842419c476b9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260807 11:35:13.093704 17105 leader_election.cc:302] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bddcb8b018745c38c842419c476b9d8; no voters:
I20260807 11:35:13.094373 17105 leader_election.cc:288] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:13.094475 17116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:13.095817 17116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [term 1 LEADER]: Becoming Leader. State: Replica: 8bddcb8b018745c38c842419c476b9d8, State: Running, Role: LEADER
I20260807 11:35:13.096509 17116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bddcb8b018745c38c842419c476b9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260807 11:35:13.097054 17105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:13.099551 17118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bddcb8b018745c38c842419c476b9d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bddcb8b018745c38c842419c476b9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } } }
I20260807 11:35:13.100296 17117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bddcb8b018745c38c842419c476b9d8. Latest consensus state: current_term: 1 leader_uuid: "8bddcb8b018745c38c842419c476b9d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bddcb8b018745c38c842419c476b9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } } }
I20260807 11:35:13.100540 17117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:13.100904 17118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:13.101440 17121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:13.113909 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:13.091713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:44499 (local address 127.0.0.1:36163)
0807 11:35:13.091928 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:13.091935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:13.092514 (+ 579us) server_negotiation.cc:407] Connection header received
0807 11:35:13.092654 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:13.092660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:13.092741 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:13.092860 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:13.094472 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:13.096070 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:13.097770 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:13.098609 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:13.099002 (+ 393us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:13.103396 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:13.103408 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:13.103416 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:13.103473 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:13.108348 (+ 4875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:13.109759 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:13.109768 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:13.109774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:13.109978 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:13.110712 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:13.110720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:13.110726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:13.111255 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:13.111608 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:13.113090 (+ 1482us) server_negotiation.cc:299] Negotiation successful
0807 11:35:13.113354 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 11:35:13.117067 17121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:13.126436 17121 catalog_manager.cc:1347] Generated new cluster ID: 0f1287896e7e494f958e296130707993
I20260807 11:35:13.126612 17121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:13.146281 17121 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:13.147704 17121 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:13.156103 17121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bddcb8b018745c38c842419c476b9d8: Generated new TSK 0
I20260807 11:35:13.157006 17121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:13.165179 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17031
I20260807 11:35:13.166772 17100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:35:13.257158 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 17031
W20260807 11:35:13.279348 17014 connection.cc:535] client connection to 127.0.0.1:36163 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36163): Connection reset by peer (error 104)
2026-08-07T11:35:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:13 dist-test-slave-wl2h krb5kdc[17141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:13 dist-test-slave-wl2h krb5kdc[17141](info): set up 2 sockets
Aug 07 11:35:13 dist-test-slave-wl2h krb5kdc[17141](info): commencing operation
krb5kdc: starting...
W20260807 11:35:14.529008 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:14 dist-test-slave-wl2h krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:35:14.883605 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:15.243546 17156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:15.243894 17156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:15.244069 17156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:15.251827 17156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:15.251952 17156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:15.252070 17156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:15.252174 17156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:15.252245 17156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:15.262802 17156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:15.264310 17156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:15.265691 17156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:15.278612 17168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:15.278700 17171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:15.278730 17169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:15.279597 17156 server_base.cc:1034] running on GCE node
I20260807 11:35:15.280472 17156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:15.281971 17156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:15.283242 17156 hybrid_clock.cc:648] HybridClock initialized: now 1786102515283207 us; error 53 us; skew 500 ppm
I20260807 11:35:15.283670 17156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:15 dist-test-slave-wl2h krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:15.287106 17156 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:35:15.288843 17156 webserver.cc:460] Webserver started at http://127.0.0.1:43155/ using document root <none> and password file <none>
I20260807 11:35:15.289577 17156 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:15.289727 17156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:15.290036 17156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:15.293227 17156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "86ef7e58182e4117bed936cdf2b4a8f4"
format_stamp: "Formatted at 2026-08-07 11:35:15 on dist-test-slave-wl2h"
server_key: "b04769d305c31ca19b49492abffd0e47"
server_key_iv: "532350683962b33b99abd4814a4a925d"
server_key_version: "encryptionkey@0"
I20260807 11:35:15.294029 17156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "86ef7e58182e4117bed936cdf2b4a8f4"
format_stamp: "Formatted at 2026-08-07 11:35:15 on dist-test-slave-wl2h"
server_key: "b04769d305c31ca19b49492abffd0e47"
server_key_iv: "532350683962b33b99abd4814a4a925d"
server_key_version: "encryptionkey@0"
I20260807 11:35:15.299609 17156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 11:35:15.303494 17181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:15.305020 17156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:35:15.305240 17156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "86ef7e58182e4117bed936cdf2b4a8f4"
format_stamp: "Formatted at 2026-08-07 11:35:15 on dist-test-slave-wl2h"
server_key: "b04769d305c31ca19b49492abffd0e47"
server_key_iv: "532350683962b33b99abd4814a4a925d"
server_key_version: "encryptionkey@0"
I20260807 11:35:15.305464 17156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:15.327502 17156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:15.332772 17156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:15.333134 17156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:15.354418 17156 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44917
I20260807 11:35:15.354451 17234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44917 every 8 connection(s)
I20260807 11:35:15.355959 17156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:15.359779 17235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:15.360008 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 17156
I20260807 11:35:15.360220 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:15.360735 24989 external_mini_cluster.cc:1428] Setting key 9a6d43f92fe9368bb163630095d7246d
I20260807 11:35:15.374939 17235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4: Bootstrap starting.
I20260807 11:35:15.377149 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:15.362770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:42475 (local address 127.0.0.1:44917)
0807 11:35:15.363637 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:15.363652 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:15.363702 (+ 50us) server_negotiation.cc:407] Connection header received
0807 11:35:15.366654 (+ 2952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:15.366684 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:15.367259 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:15.367719 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:15.368247 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:15.370124 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:15.371562 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:15.372657 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:15.372843 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:15.374281 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:15.374292 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:15.374311 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:15.375058 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:15.375163 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:15.375202 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:15.375371 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:15.375645 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 11:35:15.375811 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":397,"threads_started":1}
I20260807 11:35:15.379274 17235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:15.380774 17235 log.cc:824] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:15.384369 17235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4: No bootstrap required, opened a new log
I20260807 11:35:15.385401 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:15.393164 17235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ef7e58182e4117bed936cdf2b4a8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44917 } }
I20260807 11:35:15.393580 17235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:15.393687 17235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ef7e58182e4117bed936cdf2b4a8f4, State: Initialized, Role: FOLLOWER
I20260807 11:35:15.394297 17235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ef7e58182e4117bed936cdf2b4a8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44917 } }
I20260807 11:35:15.394534 17235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:15.394701 17235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:15.394873 17235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:15.397540 17235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ef7e58182e4117bed936cdf2b4a8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44917 } }
I20260807 11:35:15.397745 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:15.390433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:37643 (local address 127.0.0.1:44917)
0807 11:35:15.390628 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:15.390633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:15.391176 (+ 543us) server_negotiation.cc:407] Connection header received
0807 11:35:15.391415 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:15.391420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:15.391515 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:15.391640 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:15.392255 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:15.393119 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:15.394946 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:15.395466 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:15.395663 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:15.396905 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:15.396910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:15.396914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:15.396932 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:15.396969 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:15.396984 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:15.397106 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:15.397362 (+ 256us) server_negotiation.cc:299] Negotiation successful
0807 11:35:15.397417 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 11:35:15.398229 17235 leader_election.cc:302] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ef7e58182e4117bed936cdf2b4a8f4; no voters:
I20260807 11:35:15.398991 17235 leader_election.cc:288] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:15.399637 17245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:15.400298 17245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [term 1 LEADER]: Becoming Leader. State: Replica: 86ef7e58182e4117bed936cdf2b4a8f4, State: Running, Role: LEADER
I20260807 11:35:15.400933 17245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ef7e58182e4117bed936cdf2b4a8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44917 } }
I20260807 11:35:15.401321 17235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:15.404000 17247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86ef7e58182e4117bed936cdf2b4a8f4. Latest consensus state: current_term: 1 leader_uuid: "86ef7e58182e4117bed936cdf2b4a8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ef7e58182e4117bed936cdf2b4a8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44917 } } }
I20260807 11:35:15.404458 17247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:15.405058 17246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86ef7e58182e4117bed936cdf2b4a8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ef7e58182e4117bed936cdf2b4a8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44917 } } }
I20260807 11:35:15.405371 17252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:15.405445 17246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:15.410789 17252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:15.420670 17252 catalog_manager.cc:1347] Generated new cluster ID: fe6244a7984d4974bd8eb12198624bfb
I20260807 11:35:15.420814 17252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:15.427757 17252 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:15.428936 17252 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:15.440121 17252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86ef7e58182e4117bed936cdf2b4a8f4: Generated new TSK 0
I20260807 11:35:15.440900 17252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:15.446772 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17156
W20260807 11:35:15.448197 17230 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:42475
W20260807 11:35:15.449040 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17156: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:35:15.449231 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 17156
W20260807 11:35:15.471493 17137 connection.cc:535] client connection to 127.0.0.1:44917 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44917): Connection reset by peer (error 104)
2026-08-07T11:35:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2187 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:15 dist-test-slave-wl2h krb5kdc[17268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:15 dist-test-slave-wl2h krb5kdc[17268](info): set up 2 sockets
Aug 07 11:35:15 dist-test-slave-wl2h krb5kdc[17268](info): commencing operation
krb5kdc: starting...
W20260807 11:35:16.736858 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.007s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:16 dist-test-slave-wl2h krb5kdc[17268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 11:35:17.096927 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:17.455140 17281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:17.455475 17281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:17.455643 17281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:17.463397 17281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:17.463541 17281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:17.463948 17281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:17.464102 17281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:17.464178 17281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:17.475191 17281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32999
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:17.476728 17281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:17.478148 17281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:17.491297 17287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:17.491235 17288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:17.491323 17290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:17.492550 17281 server_base.cc:1034] running on GCE node
I20260807 11:35:17.493372 17281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:17.494935 17281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:17.496207 17281 hybrid_clock.cc:648] HybridClock initialized: now 1786102517496180 us; error 77 us; skew 500 ppm
I20260807 11:35:17.496556 17281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:17 dist-test-slave-wl2h krb5kdc[17268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:17.500118 17281 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 11:35:17.501870 17281 webserver.cc:460] Webserver started at http://127.0.0.1:39499/ using document root <none> and password file <none>
I20260807 11:35:17.502714 17281 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:17.502830 17281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:17.503252 17281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:17.506536 17281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "0b155b1e9a804977bb767c21e1178c87"
format_stamp: "Formatted at 2026-08-07 11:35:17 on dist-test-slave-wl2h"
server_key: "b3819575bf4191350cc0979faed4f276"
server_key_iv: "5740c76cf54bfb8eac72171d2775b088"
server_key_version: "encryptionkey@0"
I20260807 11:35:17.507452 17281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "0b155b1e9a804977bb767c21e1178c87"
format_stamp: "Formatted at 2026-08-07 11:35:17 on dist-test-slave-wl2h"
server_key: "b3819575bf4191350cc0979faed4f276"
server_key_iv: "5740c76cf54bfb8eac72171d2775b088"
server_key_version: "encryptionkey@0"
I20260807 11:35:17.513486 17281 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:35:17.517768 17298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:17.519506 17281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 11:35:17.519709 17281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "0b155b1e9a804977bb767c21e1178c87"
format_stamp: "Formatted at 2026-08-07 11:35:17 on dist-test-slave-wl2h"
server_key: "b3819575bf4191350cc0979faed4f276"
server_key_iv: "5740c76cf54bfb8eac72171d2775b088"
server_key_version: "encryptionkey@0"
I20260807 11:35:17.519923 17281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:17.526535 17281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:17.531379 17281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:17.531728 17281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:17.556211 17281 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32999
I20260807 11:35:17.556320 17354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32999 every 8 connection(s)
I20260807 11:35:17.557833 17281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:17.561965 17356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:17.564162 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 17281
I20260807 11:35:17.564366 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:17.565009 24989 external_mini_cluster.cc:1428] Setting key 99abbf5f956bbb1f26eabdb584fed85c
I20260807 11:35:17.573767 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:17.567162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:52311 (local address 127.0.0.1:32999)
0807 11:35:17.568071 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:17.568083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:17.568130 (+ 47us) server_negotiation.cc:407] Connection header received
0807 11:35:17.569796 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:17.569819 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:17.570204 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:17.570549 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:17.571877 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:17.571884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:17.571899 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:17.571919 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:17.572025 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:17.572053 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:17.572378 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:17.572523 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 11:35:17.572755 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":416,"threads_started":1}
I20260807 11:35:17.578056 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:17.578590 17356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87: Bootstrap starting.
I20260807 11:35:17.582139 17356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:17.583631 17356 log.cc:824] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:17.585700 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:17.582960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:56261 (local address 127.0.0.1:32999)
0807 11:35:17.583241 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:17.583249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:17.583570 (+ 321us) server_negotiation.cc:407] Connection header received
0807 11:35:17.583813 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:17.583821 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:17.583896 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:17.584004 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:17.585018 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:17.585024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:17.585029 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:17.585047 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:17.585094 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:17.585110 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:17.585237 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:17.585363 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 11:35:17.585427 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 11:35:17.586829 17356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87: No bootstrap required, opened a new log
I20260807 11:35:17.595794 17356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b155b1e9a804977bb767c21e1178c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } }
I20260807 11:35:17.596243 17356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:17.596328 17356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b155b1e9a804977bb767c21e1178c87, State: Initialized, Role: FOLLOWER
I20260807 11:35:17.597023 17356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b155b1e9a804977bb767c21e1178c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } }
I20260807 11:35:17.597373 17356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:17.597575 17356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:17.597787 17356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:17.600519 17356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b155b1e9a804977bb767c21e1178c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } }
I20260807 11:35:17.600996 17356 leader_election.cc:302] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b155b1e9a804977bb767c21e1178c87; no voters:
I20260807 11:35:17.601666 17356 leader_election.cc:288] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:17.601816 17367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:17.602972 17367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [term 1 LEADER]: Becoming Leader. State: Replica: 0b155b1e9a804977bb767c21e1178c87, State: Running, Role: LEADER
I20260807 11:35:17.603596 17367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b155b1e9a804977bb767c21e1178c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } }
I20260807 11:35:17.604213 17356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:17.606674 17368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b155b1e9a804977bb767c21e1178c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b155b1e9a804977bb767c21e1178c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } } }
I20260807 11:35:17.607121 17368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:17.607668 17369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b155b1e9a804977bb767c21e1178c87. Latest consensus state: current_term: 1 leader_uuid: "0b155b1e9a804977bb767c21e1178c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b155b1e9a804977bb767c21e1178c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32999 } } }
I20260807 11:35:17.608012 17373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:17.608091 17369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:17.615092 17373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:17.626309 17373 catalog_manager.cc:1347] Generated new cluster ID: aadcdef705644f9987104331d07caaa8
I20260807 11:35:17.626441 17373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:17.643143 17373 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:17.643599 17373 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:17.656306 17373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b155b1e9a804977bb767c21e1178c87: Generated new TSK 0
I20260807 11:35:17.657171 17373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:17.668072 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17281
W20260807 11:35:17.669703 17348 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:52311
W20260807 11:35:17.670559 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17281: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:35:17.670781 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 17281
2026-08-07T11:35:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:17 dist-test-slave-wl2h krb5kdc[17388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:17 dist-test-slave-wl2h krb5kdc[17388](info): set up 2 sockets
Aug 07 11:35:17 dist-test-slave-wl2h krb5kdc[17388](info): commencing operation
krb5kdc: starting...
W20260807 11:35:18.919564 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.006s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:19 dist-test-slave-wl2h krb5kdc[17388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:35:19.279443 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41321
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:41321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:19.631702 17403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:19.632035 17403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:19.632244 17403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:19.639891 17403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:19.640020 17403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:19.640146 17403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:19.640242 17403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:19.640306 17403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:19.650606 17403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:41321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:41321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:19.652082 17403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:19.653507 17403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:19.665735 17409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:19.665695 17411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:19.665783 17408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:19.666735 17403 server_base.cc:1034] running on GCE node
I20260807 11:35:19.667587 17403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:19.668990 17403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:19.670189 17403 hybrid_clock.cc:648] HybridClock initialized: now 1786102519670135 us; error 58 us; skew 500 ppm
I20260807 11:35:19.670507 17403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:19 dist-test-slave-wl2h krb5kdc[17388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102519, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:19.679376 17403 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:35:19.681027 17403 webserver.cc:460] Webserver started at http://127.24.103.126:33481/ using document root <none> and password file <none>
I20260807 11:35:19.681763 17403 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:19.681874 17403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:19.682242 17403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:19.685567 17403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "177b5f64ccf040eb81fa8b4c3b8dbf41"
format_stamp: "Formatted at 2026-08-07 11:35:19 on dist-test-slave-wl2h"
server_key: "33f20b235e56d9dd03f0212cbd6c9522"
server_key_iv: "45fe74b471c1e24b29b224290b1e1cc3"
server_key_version: "encryptionkey@0"
I20260807 11:35:19.686370 17403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "177b5f64ccf040eb81fa8b4c3b8dbf41"
format_stamp: "Formatted at 2026-08-07 11:35:19 on dist-test-slave-wl2h"
server_key: "33f20b235e56d9dd03f0212cbd6c9522"
server_key_iv: "45fe74b471c1e24b29b224290b1e1cc3"
server_key_version: "encryptionkey@0"
I20260807 11:35:19.692087 17403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 11:35:19.695902 17419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:19.697371 17403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:35:19.697564 17403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "177b5f64ccf040eb81fa8b4c3b8dbf41"
format_stamp: "Formatted at 2026-08-07 11:35:19 on dist-test-slave-wl2h"
server_key: "33f20b235e56d9dd03f0212cbd6c9522"
server_key_iv: "45fe74b471c1e24b29b224290b1e1cc3"
server_key_version: "encryptionkey@0"
I20260807 11:35:19.697767 17403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:19.722312 17403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:19.726775 17403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:19.727088 17403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:19.745918 17403 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:41321
I20260807 11:35:19.745941 17482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:41321 every 8 connection(s)
I20260807 11:35:19.747365 17403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:19.751303 17483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:19.754331 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 17403
I20260807 11:35:19.754525 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:19.755129 24989 external_mini_cluster.cc:1428] Setting key 19d82109747cf3f729da0b069746bf08
I20260807 11:35:19.765463 17483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41: Bootstrap starting.
I20260807 11:35:19.769369 17483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:19.770495 17483 log.cc:824] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41: Log is configured to *not* fsync() on all Append() calls
Aug 07 11:35:19 dist-test-slave-wl2h krb5kdc[17388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:35:19.773847 17483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41: No bootstrap required, opened a new log
I20260807 11:35:19.781162 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:19.757158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47047 (local address 127.24.103.126:41321)
0807 11:35:19.757993 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:19.758010 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:19.758075 (+ 65us) server_negotiation.cc:407] Connection header received
0807 11:35:19.760458 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:19.760490 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:19.761060 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:19.761517 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:19.762868 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:19.764871 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:19.766292 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:19.767404 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:19.767611 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:19.772402 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:19.772410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:19.772425 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:19.772464 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:19.775997 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:19.777040 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:19.777046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:19.777053 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:19.777217 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:19.777750 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:19.777754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:19.777756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:19.778327 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:19.778591 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:19.779272 (+ 681us) server_negotiation.cc:299] Negotiation successful
0807 11:35:19.779675 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":342,"threads_started":1}
I20260807 11:35:19.783310 17483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b5f64ccf040eb81fa8b4c3b8dbf41" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41321 } }
I20260807 11:35:19.783931 17483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:19.784093 17483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 177b5f64ccf040eb81fa8b4c3b8dbf41, State: Initialized, Role: FOLLOWER
I20260807 11:35:19.784816 17483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b5f64ccf040eb81fa8b4c3b8dbf41" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41321 } }
I20260807 11:35:19.785167 17483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:19.785341 17483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:19.785549 17483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:19.785929 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:19.788993 17483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b5f64ccf040eb81fa8b4c3b8dbf41" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41321 } }
I20260807 11:35:19.789718 17483 leader_election.cc:302] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 177b5f64ccf040eb81fa8b4c3b8dbf41; no voters:
I20260807 11:35:19.790530 17483 leader_election.cc:288] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:19.790637 17494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:19.792872 17494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [term 1 LEADER]: Becoming Leader. State: Replica: 177b5f64ccf040eb81fa8b4c3b8dbf41, State: Running, Role: LEADER
I20260807 11:35:19.793097 17483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:19.793601 17494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b5f64ccf040eb81fa8b4c3b8dbf41" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41321 } }
I20260807 11:35:19.796680 17497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 177b5f64ccf040eb81fa8b4c3b8dbf41. Latest consensus state: current_term: 1 leader_uuid: "177b5f64ccf040eb81fa8b4c3b8dbf41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b5f64ccf040eb81fa8b4c3b8dbf41" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41321 } } }
I20260807 11:35:19.797067 17497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:19.797711 17496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "177b5f64ccf040eb81fa8b4c3b8dbf41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b5f64ccf040eb81fa8b4c3b8dbf41" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 41321 } } }
I20260807 11:35:19.798069 17496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:19.815279 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:19.792319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37219 (local address 127.24.103.126:41321)
0807 11:35:19.792553 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:19.792562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:19.793086 (+ 524us) server_negotiation.cc:407] Connection header received
0807 11:35:19.793389 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:19.793395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:19.793483 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:19.793606 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:19.795693 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:19.797008 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:19.799181 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:19.800123 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:19.800244 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:19.804648 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:19.804658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:19.804667 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:19.804724 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:19.809509 (+ 4785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:19.810964 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:19.810973 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:19.810980 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:19.811238 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:19.811952 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:19.811960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:19.811966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:19.812596 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:19.812948 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:19.814507 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0807 11:35:19.814726 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 11:35:19.816993 17503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:19.823236 17503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:19.834124 17503 catalog_manager.cc:1347] Generated new cluster ID: 29a0953e6726445ab7c398d1d0dcecc3
I20260807 11:35:19.834345 17503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:19.848533 17503 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:19.850539 17503 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:19.862920 17503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 177b5f64ccf040eb81fa8b4c3b8dbf41: Generated new TSK 0
I20260807 11:35:19.863903 17503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:19.872915 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17403
I20260807 11:35:19.874277 17478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:35:19.968344 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 17403
W20260807 11:35:19.991375 17386 connection.cc:535] client connection to 127.24.103.126:41321 recv error: Network error: failed to read from TLS socket (remote: 127.24.103.126:41321): Connection reset by peer (error 104)
2026-08-07T11:35:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:20 dist-test-slave-wl2h krb5kdc[17522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:20 dist-test-slave-wl2h krb5kdc[17522](info): set up 2 sockets
Aug 07 11:35:20 dist-test-slave-wl2h krb5kdc[17522](info): commencing operation
krb5kdc: starting...
W20260807 11:35:21.201321 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.002s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:21 dist-test-slave-wl2h krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:35:21.552752 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:45871
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:45871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:21.909003 17537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:21.909304 17537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:21.909471 17537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:21.917151 17537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:21.917271 17537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:21.917378 17537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:21.917484 17537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:21.917563 17537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:21.927882 17537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:45871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:45871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:21.929278 17537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:21.930636 17537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:21.943425 17543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:21.943372 17545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:21.943432 17542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:21.944448 17537 server_base.cc:1034] running on GCE node
I20260807 11:35:21.945251 17537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:21.946743 17537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:21.947974 17537 hybrid_clock.cc:648] HybridClock initialized: now 1786102521947933 us; error 55 us; skew 500 ppm
I20260807 11:35:21.948349 17537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:21 dist-test-slave-wl2h krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:21.951720 17537 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:35:21.953231 17537 webserver.cc:460] Webserver started at http://127.24.103.126:39089/ using document root <none> and password file <none>
I20260807 11:35:21.953948 17537 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:21.954105 17537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:21.954417 17537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:21.957561 17537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "01915e6caa5c4fa79755545694d87d74"
format_stamp: "Formatted at 2026-08-07 11:35:21 on dist-test-slave-wl2h"
server_key: "ffb1876e11b9915df48bf5c52cc7b6db"
server_key_iv: "23fc50addf19c51959ffbd7b903e330b"
server_key_version: "encryptionkey@0"
I20260807 11:35:21.958336 17537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "01915e6caa5c4fa79755545694d87d74"
format_stamp: "Formatted at 2026-08-07 11:35:21 on dist-test-slave-wl2h"
server_key: "ffb1876e11b9915df48bf5c52cc7b6db"
server_key_iv: "23fc50addf19c51959ffbd7b903e330b"
server_key_version: "encryptionkey@0"
I20260807 11:35:21.963737 17537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 11:35:21.967314 17552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:21.968870 17537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:35:21.969080 17537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "01915e6caa5c4fa79755545694d87d74"
format_stamp: "Formatted at 2026-08-07 11:35:21 on dist-test-slave-wl2h"
server_key: "ffb1876e11b9915df48bf5c52cc7b6db"
server_key_iv: "23fc50addf19c51959ffbd7b903e330b"
server_key_version: "encryptionkey@0"
I20260807 11:35:21.969377 17537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:21.993350 17537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:21.997982 17537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:21.998312 17537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:22.018407 17537 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:45871
I20260807 11:35:22.018440 17610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:45871 every 8 connection(s)
I20260807 11:35:22.019886 17537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:22.023746 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:22.028282 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 17537
I20260807 11:35:22.028534 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:22.029138 24989 external_mini_cluster.cc:1428] Setting key d59bad443b93bb77dea1dfef06ed9cf1
I20260807 11:35:22.036125 17611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74: Bootstrap starting.
I20260807 11:35:22.039935 17611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:22.041337 17611 log.cc:824] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:22.044749 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:22.031423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52011 (local address 127.24.103.126:45871)
0807 11:35:22.032326 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:22.032337 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:22.032382 (+ 45us) server_negotiation.cc:407] Connection header received
0807 11:35:22.034447 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:22.034469 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:22.034888 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:22.035296 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:22.035982 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:22.037437 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:22.039441 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:22.040232 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:22.040395 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:22.041778 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:22.041786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:22.041800 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:22.042686 (+ 886us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:22.042800 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:22.042839 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:22.043052 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:22.043566 (+ 514us) server_negotiation.cc:299] Negotiation successful
0807 11:35:22.043777 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":406,"threads_started":1}
I20260807 11:35:22.044845 17611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74: No bootstrap required, opened a new log
I20260807 11:35:22.048839 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:22.055109 17611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01915e6caa5c4fa79755545694d87d74" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45871 } }
I20260807 11:35:22.055670 17611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:22.055788 17611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01915e6caa5c4fa79755545694d87d74, State: Initialized, Role: FOLLOWER
I20260807 11:35:22.056456 17611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01915e6caa5c4fa79755545694d87d74" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45871 } }
I20260807 11:35:22.056787 17611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:22.056948 17611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:22.057134 17611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:22.060005 17611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01915e6caa5c4fa79755545694d87d74" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45871 } }
I20260807 11:35:22.060568 17611 leader_election.cc:302] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01915e6caa5c4fa79755545694d87d74; no voters:
I20260807 11:35:22.061378 17611 leader_election.cc:288] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:22.061631 17622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:22.062458 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:22.054467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53269 (local address 127.24.103.126:45871)
0807 11:35:22.054676 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:22.054684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:22.055243 (+ 559us) server_negotiation.cc:407] Connection header received
0807 11:35:22.055410 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:22.055415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:22.055506 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:22.055617 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:22.056392 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:22.057576 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:22.058988 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:22.059704 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:22.059850 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:22.061116 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:22.061125 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:22.061143 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:22.061180 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:22.061276 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:22.061300 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:22.061448 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:22.061883 (+ 435us) server_negotiation.cc:299] Negotiation successful
0807 11:35:22.061970 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:35:22.065052 17611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:22.068540 17622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [term 1 LEADER]: Becoming Leader. State: Replica: 01915e6caa5c4fa79755545694d87d74, State: Running, Role: LEADER
I20260807 11:35:22.069698 17622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01915e6caa5c4fa79755545694d87d74" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45871 } }
I20260807 11:35:22.073100 17623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01915e6caa5c4fa79755545694d87d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01915e6caa5c4fa79755545694d87d74" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45871 } } }
I20260807 11:35:22.073534 17623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:22.073325 17624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01915e6caa5c4fa79755545694d87d74. Latest consensus state: current_term: 1 leader_uuid: "01915e6caa5c4fa79755545694d87d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01915e6caa5c4fa79755545694d87d74" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 45871 } } }
I20260807 11:35:22.074158 17624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74 [sys.catalog]: This master's current role is: LEADER
W20260807 11:35:22.079938 17640 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:35:22.080096 17640 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:35:22.080287 17641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:22.082525 17641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:22.090850 17641 catalog_manager.cc:1347] Generated new cluster ID: b4d5bee8b765411784df9a664c5d8e7e
I20260807 11:35:22.090986 17641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:22.103909 17641 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:22.105800 17641 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:22.119204 17641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01915e6caa5c4fa79755545694d87d74: Generated new TSK 0
I20260807 11:35:22.120121 17641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:22.131600 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17537
W20260807 11:35:22.132975 17606 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52011
W20260807 11:35:22.133791 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17537: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:35:22.133952 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 17537
W20260807 11:35:22.157013 17520 connection.cc:535] client connection to 127.24.103.126:45871 recv error: Network error: failed to read from TLS socket (remote: 127.24.103.126:45871): Connection reset by peer (error 104)
2026-08-07T11:35:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2165 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:22 dist-test-slave-wl2h krb5kdc[17648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:22 dist-test-slave-wl2h krb5kdc[17648](info): set up 2 sockets
Aug 07 11:35:22 dist-test-slave-wl2h krb5kdc[17648](info): commencing operation
krb5kdc: starting...
W20260807 11:35:23.388994 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:23 dist-test-slave-wl2h krb5kdc[17648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:35:23.756644 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34299
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:34299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:24.123247 17661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:24.123549 17661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:24.123719 17661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:24.131181 17661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:24.131300 17661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:24.131392 17661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:24.131501 17661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:24.131565 17661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:24.142553 17661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:34299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:34299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:24.144049 17661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:24.145466 17661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:24.158193 17666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:24.158245 17667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:24.158269 17669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:24.159118 17661 server_base.cc:1034] running on GCE node
I20260807 11:35:24.159920 17661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:24.161414 17661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:24.162714 17661 hybrid_clock.cc:648] HybridClock initialized: now 1786102524162669 us; error 41 us; skew 500 ppm
I20260807 11:35:24.163232 17661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:24 dist-test-slave-wl2h krb5kdc[17648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102524, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:24.166558 17661 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:35:24.168352 17661 webserver.cc:460] Webserver started at http://127.24.103.126:36917/ using document root <none> and password file <none>
I20260807 11:35:24.169112 17661 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:24.169273 17661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:24.169605 17661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:24.172890 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "c479eec15f3f476ca1791b89a3e4b9c8"
format_stamp: "Formatted at 2026-08-07 11:35:24 on dist-test-slave-wl2h"
server_key: "406edc79691fe8cfdcaea6cd2456031e"
server_key_iv: "d37927e27328845d3558597f0cf49626"
server_key_version: "encryptionkey@0"
I20260807 11:35:24.173772 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "c479eec15f3f476ca1791b89a3e4b9c8"
format_stamp: "Formatted at 2026-08-07 11:35:24 on dist-test-slave-wl2h"
server_key: "406edc79691fe8cfdcaea6cd2456031e"
server_key_iv: "d37927e27328845d3558597f0cf49626"
server_key_version: "encryptionkey@0"
I20260807 11:35:24.179365 17661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 11:35:24.183175 17677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:24.184604 17661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:35:24.184823 17661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "c479eec15f3f476ca1791b89a3e4b9c8"
format_stamp: "Formatted at 2026-08-07 11:35:24 on dist-test-slave-wl2h"
server_key: "406edc79691fe8cfdcaea6cd2456031e"
server_key_iv: "d37927e27328845d3558597f0cf49626"
server_key_version: "encryptionkey@0"
I20260807 11:35:24.185055 17661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:24.191205 17661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:24.195678 17661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:24.196007 17661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:24.216501 17661 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:34299
I20260807 11:35:24.216601 17733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:34299 every 8 connection(s)
I20260807 11:35:24.217981 17661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:24.221880 17734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:24.223924 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 17661
I20260807 11:35:24.224120 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:24.224670 24989 external_mini_cluster.cc:1428] Setting key 6a44f6534335c2e5f6848ce70e7c2934
I20260807 11:35:24.232759 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:24.226639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51157 (local address 127.24.103.126:34299)
0807 11:35:24.227543 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:24.227555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:24.227608 (+ 53us) server_negotiation.cc:407] Connection header received
0807 11:35:24.229262 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:24.229287 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:24.229636 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:24.229956 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:24.231139 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:24.231145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:24.231157 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:24.231175 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:24.231234 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:24.231254 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:24.231438 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:24.231546 (+ 108us) server_negotiation.cc:299] Negotiation successful
0807 11:35:24.231752 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":452,"threads_started":1}
I20260807 11:35:24.235859 17734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8: Bootstrap starting.
I20260807 11:35:24.236454 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:24.238946 17734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:24.240139 17734 log.cc:824] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:24.243247 17734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8: No bootstrap required, opened a new log
I20260807 11:35:24.243528 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:24.240930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33933 (local address 127.24.103.126:34299)
0807 11:35:24.241150 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:24.241156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:24.241476 (+ 320us) server_negotiation.cc:407] Connection header received
0807 11:35:24.241669 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:24.241675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:24.241737 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:24.241844 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:24.242777 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:24.242783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:24.242789 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:24.242810 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:24.242847 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:24.242860 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:24.242982 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:24.243116 (+ 134us) server_negotiation.cc:299] Negotiation successful
0807 11:35:24.243179 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 11:35:24.251559 17734 raft_consensus.cc:348] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c479eec15f3f476ca1791b89a3e4b9c8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34299 } }
I20260807 11:35:24.251981 17734 raft_consensus.cc:374] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:24.252112 17734 raft_consensus.cc:729] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c479eec15f3f476ca1791b89a3e4b9c8, State: Initialized, Role: FOLLOWER
I20260807 11:35:24.252738 17734 consensus_queue.cc:260] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c479eec15f3f476ca1791b89a3e4b9c8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34299 } }
I20260807 11:35:24.252980 17734 raft_consensus.cc:388] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:24.253139 17734 raft_consensus.cc:482] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:24.253334 17734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:24.256232 17734 raft_consensus.cc:504] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c479eec15f3f476ca1791b89a3e4b9c8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34299 } }
I20260807 11:35:24.256759 17734 leader_election.cc:302] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c479eec15f3f476ca1791b89a3e4b9c8; no voters:
I20260807 11:35:24.257375 17734 leader_election.cc:288] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:24.257535 17745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:24.258265 17745 raft_consensus.cc:686] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [term 1 LEADER]: Becoming Leader. State: Replica: c479eec15f3f476ca1791b89a3e4b9c8, State: Running, Role: LEADER
I20260807 11:35:24.258808 17745 consensus_queue.cc:237] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c479eec15f3f476ca1791b89a3e4b9c8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34299 } }
I20260807 11:35:24.259917 17734 sys_catalog.cc:564] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:24.262948 17747 sys_catalog.cc:455] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c479eec15f3f476ca1791b89a3e4b9c8. Latest consensus state: current_term: 1 leader_uuid: "c479eec15f3f476ca1791b89a3e4b9c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c479eec15f3f476ca1791b89a3e4b9c8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34299 } } }
I20260807 11:35:24.263440 17747 sys_catalog.cc:458] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:24.266026 17746 sys_catalog.cc:455] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c479eec15f3f476ca1791b89a3e4b9c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c479eec15f3f476ca1791b89a3e4b9c8" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 34299 } } }
I20260807 11:35:24.266487 17746 sys_catalog.cc:458] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:24.270375 17754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:24.277009 17754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:24.288661 17754 catalog_manager.cc:1347] Generated new cluster ID: 22e58bfcc7a1469a99d05c19c25cbaac
I20260807 11:35:24.289052 17754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:24.310755 17754 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:24.311373 17754 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:24.322690 17754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c479eec15f3f476ca1791b89a3e4b9c8: Generated new TSK 0
I20260807 11:35:24.323853 17754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:24.340337 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17661
W20260807 11:35:24.341562 17729 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51157
W20260807 11:35:24.342367 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17661: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:35:24.342533 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 17661
2026-08-07T11:35:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:24 dist-test-slave-wl2h krb5kdc[17773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:24 dist-test-slave-wl2h krb5kdc[17773](info): set up 2 sockets
Aug 07 11:35:24 dist-test-slave-wl2h krb5kdc[17773](info): commencing operation
krb5kdc: starting...
W20260807 11:35:25.593873 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:25 dist-test-slave-wl2h krb5kdc[17773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:35:25.961987 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43241
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:43241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:26.328821 17789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:26.329119 17789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:26.329335 17789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:26.336664 17789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:26.336774 17789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:26.336874 17789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:26.336985 17789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:26.337059 17789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:26.346946 17789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:43241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:26.348316 17789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:26.349644 17789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:26.362144 17795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:26.362093 17796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:26.362046 17798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:26.363054 17789 server_base.cc:1034] running on GCE node
I20260807 11:35:26.363866 17789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:26.365283 17789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:26.366508 17789 hybrid_clock.cc:648] HybridClock initialized: now 1786102526366440 us; error 42 us; skew 500 ppm
I20260807 11:35:26.366892 17789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:26 dist-test-slave-wl2h krb5kdc[17773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102526, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:26.375211 17789 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:35:26.377178 17789 webserver.cc:460] Webserver started at http://127.24.103.126:37313/ using document root <none> and password file <none>
I20260807 11:35:26.377967 17789 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:26.378131 17789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:26.378477 17789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:26.381824 17789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "66e82dd8f541451eb2917ecce2e9c674"
format_stamp: "Formatted at 2026-08-07 11:35:26 on dist-test-slave-wl2h"
server_key: "f8aff4b28e17b553f4466c531784ae0e"
server_key_iv: "6e4b33df8fbca34eff4d7f2e9259554c"
server_key_version: "encryptionkey@0"
I20260807 11:35:26.382702 17789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "66e82dd8f541451eb2917ecce2e9c674"
format_stamp: "Formatted at 2026-08-07 11:35:26 on dist-test-slave-wl2h"
server_key: "f8aff4b28e17b553f4466c531784ae0e"
server_key_iv: "6e4b33df8fbca34eff4d7f2e9259554c"
server_key_version: "encryptionkey@0"
I20260807 11:35:26.388459 17789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 11:35:26.392247 17807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:26.393641 17789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:35:26.393862 17789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "66e82dd8f541451eb2917ecce2e9c674"
format_stamp: "Formatted at 2026-08-07 11:35:26 on dist-test-slave-wl2h"
server_key: "f8aff4b28e17b553f4466c531784ae0e"
server_key_iv: "6e4b33df8fbca34eff4d7f2e9259554c"
server_key_version: "encryptionkey@0"
I20260807 11:35:26.394060 17789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:26.405321 17789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:26.409704 17789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:26.409991 17789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:26.429009 17789 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:43241
I20260807 11:35:26.429044 17866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:43241 every 8 connection(s)
I20260807 11:35:26.430387 17789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:26.434015 17867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:26.438256 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 17789
I20260807 11:35:26.438448 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:26.439047 24989 external_mini_cluster.cc:1428] Setting key d285de98a43d9f79de6c46793dae8424
I20260807 11:35:26.447664 17867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674: Bootstrap starting.
I20260807 11:35:26.451646 17867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:26.453047 17867 log.cc:824] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674: Log is configured to *not* fsync() on all Append() calls
Aug 07 11:35:26 dist-test-slave-wl2h krb5kdc[17773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:35:26.456125 17867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674: No bootstrap required, opened a new log
I20260807 11:35:26.462312 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:26.440872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58399 (local address 127.24.103.126:43241)
0807 11:35:26.441651 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:26.441662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:26.441709 (+ 47us) server_negotiation.cc:407] Connection header received
0807 11:35:26.443299 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:26.443320 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:26.443723 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:26.444041 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:26.445990 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:26.447371 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:26.449241 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:26.449940 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:26.450073 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:26.454965 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:26.454973 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:26.454986 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:26.455076 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:26.458217 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:26.458933 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:26.458940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:26.458945 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:26.459179 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:26.459589 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:26.459595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:26.459598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:26.460086 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:26.460425 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:26.460880 (+ 455us) server_negotiation.cc:299] Negotiation successful
0807 11:35:26.461311 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":370,"threads_started":1}
I20260807 11:35:26.464486 17867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e82dd8f541451eb2917ecce2e9c674" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43241 } }
I20260807 11:35:26.464941 17867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:26.465044 17867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e82dd8f541451eb2917ecce2e9c674, State: Initialized, Role: FOLLOWER
I20260807 11:35:26.465696 17867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e82dd8f541451eb2917ecce2e9c674" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43241 } }
I20260807 11:35:26.465972 17867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:26.466146 17867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:26.466212 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:26.466329 17867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:26.468763 17867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e82dd8f541451eb2917ecce2e9c674" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43241 } }
I20260807 11:35:26.469328 17867 leader_election.cc:302] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66e82dd8f541451eb2917ecce2e9c674; no voters:
I20260807 11:35:26.470016 17867 leader_election.cc:288] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:26.470129 17877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:26.471498 17877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [term 1 LEADER]: Becoming Leader. State: Replica: 66e82dd8f541451eb2917ecce2e9c674, State: Running, Role: LEADER
I20260807 11:35:26.472199 17877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e82dd8f541451eb2917ecce2e9c674" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43241 } }
I20260807 11:35:26.473644 17867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:26.475220 17879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66e82dd8f541451eb2917ecce2e9c674. Latest consensus state: current_term: 1 leader_uuid: "66e82dd8f541451eb2917ecce2e9c674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e82dd8f541451eb2917ecce2e9c674" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43241 } } }
I20260807 11:35:26.475657 17879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:26.476145 17878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66e82dd8f541451eb2917ecce2e9c674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e82dd8f541451eb2917ecce2e9c674" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43241 } } }
I20260807 11:35:26.476379 17878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:26.480273 17885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:26.486399 17885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:26.495616 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:26.470942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60211 (local address 127.24.103.126:43241)
0807 11:35:26.471289 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:26.471296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:26.472499 (+ 1203us) server_negotiation.cc:407] Connection header received
0807 11:35:26.472761 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:26.472766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:26.472842 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:26.472947 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:26.480441 (+ 7494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:26.481592 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:26.483597 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:26.484328 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:26.484469 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:26.488018 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:26.488025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:26.488030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:26.488072 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:26.492235 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:26.493137 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:26.493142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:26.493145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:26.493302 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:26.493807 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:26.493812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:26.493817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:26.494237 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:26.494454 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:26.495131 (+ 677us) server_negotiation.cc:299] Negotiation successful
0807 11:35:26.495363 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260807 11:35:26.498488 17885 catalog_manager.cc:1347] Generated new cluster ID: 2eb00d4c5a3e4949910ee692566deb32
I20260807 11:35:26.498622 17885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:26.516938 17885 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:26.518851 17885 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:26.528766 17885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66e82dd8f541451eb2917ecce2e9c674: Generated new TSK 0
I20260807 11:35:26.529870 17885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:26.537535 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17789
I20260807 11:35:26.539242 17861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:35:26.630210 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 17789
2026-08-07T11:35:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:26 dist-test-slave-wl2h krb5kdc[17904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:26 dist-test-slave-wl2h krb5kdc[17904](info): set up 2 sockets
Aug 07 11:35:26 dist-test-slave-wl2h krb5kdc[17904](info): commencing operation
krb5kdc: starting...
W20260807 11:35:27.876412 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:28 dist-test-slave-wl2h krb5kdc[17904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:35:28.248013 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43505
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:43505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:28.608145 17918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:28.608454 17918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:28.608628 17918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:28.616225 17918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:28.616345 17918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:28.616456 17918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:28.616566 17918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:28.616633 17918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:28.626894 17918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:43505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:43505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:28.628365 17918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:28.629791 17918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:28.643229 17923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:28.643261 17926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:28.643229 17924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:28.644109 17918 server_base.cc:1034] running on GCE node
I20260807 11:35:28.644964 17918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:28.646490 17918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:28.647750 17918 hybrid_clock.cc:648] HybridClock initialized: now 1786102528647698 us; error 43 us; skew 500 ppm
I20260807 11:35:28.648185 17918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:28 dist-test-slave-wl2h krb5kdc[17904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102528, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:28.651765 17918 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:35:28.653597 17918 webserver.cc:460] Webserver started at http://127.24.103.126:34709/ using document root <none> and password file <none>
I20260807 11:35:28.654596 17918 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:28.654784 17918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:28.655193 17918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:28.658461 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "f73d90f11a85438ab93097bca6d5e86c"
format_stamp: "Formatted at 2026-08-07 11:35:28 on dist-test-slave-wl2h"
server_key: "da7df89b826c418ba37884c23317b2e6"
server_key_iv: "8e9f00a8fd80e4f0e1853d66e0a8f7d3"
server_key_version: "encryptionkey@0"
I20260807 11:35:28.659374 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "f73d90f11a85438ab93097bca6d5e86c"
format_stamp: "Formatted at 2026-08-07 11:35:28 on dist-test-slave-wl2h"
server_key: "da7df89b826c418ba37884c23317b2e6"
server_key_iv: "8e9f00a8fd80e4f0e1853d66e0a8f7d3"
server_key_version: "encryptionkey@0"
I20260807 11:35:28.665163 17918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:35:28.669535 17933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:28.671221 17918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:35:28.671474 17918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "f73d90f11a85438ab93097bca6d5e86c"
format_stamp: "Formatted at 2026-08-07 11:35:28 on dist-test-slave-wl2h"
server_key: "da7df89b826c418ba37884c23317b2e6"
server_key_iv: "8e9f00a8fd80e4f0e1853d66e0a8f7d3"
server_key_version: "encryptionkey@0"
I20260807 11:35:28.671715 17918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:28.698030 17918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:28.702499 17918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:28.702827 17918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:28.722958 17918 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:43505
I20260807 11:35:28.722992 17989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:43505 every 8 connection(s)
I20260807 11:35:28.724424 17918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:28.725190 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 17918
I20260807 11:35:28.725411 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:28.726015 24989 external_mini_cluster.cc:1428] Setting key f057d2b1a8466ba18952aee8193d98cc
I20260807 11:35:28.729607 17990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:28.741616 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:28.728063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33467 (local address 127.24.103.126:43505)
0807 11:35:28.728897 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:28.728912 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:28.728961 (+ 49us) server_negotiation.cc:407] Connection header received
0807 11:35:28.731342 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:28.731368 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:28.731857 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:28.732303 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:28.732874 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:28.734846 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:28.736341 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:28.737342 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:28.737544 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:28.738825 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:28.738837 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:28.738854 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:28.739672 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:28.739769 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:28.739806 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:28.739943 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:28.740267 (+ 324us) server_negotiation.cc:299] Negotiation successful
0807 11:35:28.740532 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":368,"threads_started":1}
I20260807 11:35:28.743535 17990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c: Bootstrap starting.
I20260807 11:35:28.745687 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:28.747121 17990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:28.748131 17990 log.cc:824] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:28.751864 17990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c: No bootstrap required, opened a new log
I20260807 11:35:28.758883 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:28.751515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54865 (local address 127.24.103.126:43505)
0807 11:35:28.751790 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:28.751796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:28.752225 (+ 429us) server_negotiation.cc:407] Connection header received
0807 11:35:28.752454 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:28.752459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:28.752535 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:28.752620 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:28.753334 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:28.754117 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:28.756159 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:28.756655 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:28.756760 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:28.758037 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:28.758042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:28.758046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:28.758064 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:28.758116 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:28.758136 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:28.758238 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:28.758390 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 11:35:28.758476 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 11:35:28.762038 17990 raft_consensus.cc:348] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d90f11a85438ab93097bca6d5e86c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43505 } }
I20260807 11:35:28.762420 17990 raft_consensus.cc:374] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:28.762575 17990 raft_consensus.cc:729] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73d90f11a85438ab93097bca6d5e86c, State: Initialized, Role: FOLLOWER
I20260807 11:35:28.763228 17990 consensus_queue.cc:260] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d90f11a85438ab93097bca6d5e86c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43505 } }
I20260807 11:35:28.763494 17990 raft_consensus.cc:388] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:28.763677 17990 raft_consensus.cc:482] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:28.763893 17990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:28.766151 17990 raft_consensus.cc:504] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d90f11a85438ab93097bca6d5e86c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43505 } }
I20260807 11:35:28.766660 17990 leader_election.cc:302] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f73d90f11a85438ab93097bca6d5e86c; no voters:
I20260807 11:35:28.767328 17990 leader_election.cc:288] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:28.767481 18002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:28.768179 18002 raft_consensus.cc:686] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [term 1 LEADER]: Becoming Leader. State: Replica: f73d90f11a85438ab93097bca6d5e86c, State: Running, Role: LEADER
I20260807 11:35:28.768784 18002 consensus_queue.cc:237] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d90f11a85438ab93097bca6d5e86c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43505 } }
I20260807 11:35:28.769495 17990 sys_catalog.cc:564] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:28.771615 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f73d90f11a85438ab93097bca6d5e86c. Latest consensus state: current_term: 1 leader_uuid: "f73d90f11a85438ab93097bca6d5e86c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d90f11a85438ab93097bca6d5e86c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43505 } } }
I20260807 11:35:28.771994 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:28.773038 18008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:28.772776 18003 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f73d90f11a85438ab93097bca6d5e86c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d90f11a85438ab93097bca6d5e86c" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 43505 } } }
I20260807 11:35:28.774022 18003 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:28.779098 18008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:28.789413 18008 catalog_manager.cc:1347] Generated new cluster ID: 13d0094dbd3e42b1aca621482589a18b
I20260807 11:35:28.789552 18008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:28.824232 18008 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:28.825870 18008 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:28.839718 18008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f73d90f11a85438ab93097bca6d5e86c: Generated new TSK 0
I20260807 11:35:28.840582 18008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:28.853315 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17918
W20260807 11:35:28.854600 17985 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33467
W20260807 11:35:28.855372 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 17918: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:35:28.855528 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 17918
2026-08-07T11:35:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:28 dist-test-slave-wl2h krb5kdc[18024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:28 dist-test-slave-wl2h krb5kdc[18024](info): set up 2 sockets
Aug 07 11:35:28 dist-test-slave-wl2h krb5kdc[18024](info): commencing operation
krb5kdc: starting...
W20260807 11:35:30.128875 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:30 dist-test-slave-wl2h krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:35:30.488747 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39037
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:39037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:30.855551 18041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:30.855901 18041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:30.856081 18041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:30.863577 18041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:30.863740 18041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:30.863853 18041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:30.863968 18041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:30.864058 18041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:30.874764 18041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:39037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:39037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:30.876262 18041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:30.877700 18041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:30.891251 18046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:30.891175 18049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:30.891218 18047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:30.892206 18041 server_base.cc:1034] running on GCE node
I20260807 11:35:30.893066 18041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:30.894589 18041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:30.895802 18041 hybrid_clock.cc:648] HybridClock initialized: now 1786102530895751 us; error 39 us; skew 500 ppm
I20260807 11:35:30.896186 18041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:30 dist-test-slave-wl2h krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102530, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:30.899487 18041 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:35:30.901036 18041 webserver.cc:460] Webserver started at http://127.24.103.126:37313/ using document root <none> and password file <none>
I20260807 11:35:30.901701 18041 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:30.901804 18041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:30.902066 18041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:30.905122 18041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "ce933b1fd02e46b38c705c3698920b57"
format_stamp: "Formatted at 2026-08-07 11:35:30 on dist-test-slave-wl2h"
server_key: "d58d001ccc14c467d6ba76efaae38cab"
server_key_iv: "3cf69aab5d98f1bf7f3024b0099287f2"
server_key_version: "encryptionkey@0"
I20260807 11:35:30.905870 18041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "ce933b1fd02e46b38c705c3698920b57"
format_stamp: "Formatted at 2026-08-07 11:35:30 on dist-test-slave-wl2h"
server_key: "d58d001ccc14c467d6ba76efaae38cab"
server_key_iv: "3cf69aab5d98f1bf7f3024b0099287f2"
server_key_version: "encryptionkey@0"
I20260807 11:35:30.911343 18041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 11:35:30.915350 18057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:30.916795 18041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 11:35:30.916958 18041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "ce933b1fd02e46b38c705c3698920b57"
format_stamp: "Formatted at 2026-08-07 11:35:30 on dist-test-slave-wl2h"
server_key: "d58d001ccc14c467d6ba76efaae38cab"
server_key_iv: "3cf69aab5d98f1bf7f3024b0099287f2"
server_key_version: "encryptionkey@0"
I20260807 11:35:30.917164 18041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:30.923666 18041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:30.928629 18041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:30.928920 18041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:30.950362 18041 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:39037
I20260807 11:35:30.950408 18115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:39037 every 8 connection(s)
I20260807 11:35:30.951947 18041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:30.955437 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 18041
I20260807 11:35:30.955636 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:30.956243 24989 external_mini_cluster.cc:1428] Setting key ffa72a36e63eee4dfc905cc580c9a681
I20260807 11:35:30.956393 18116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:30.965221 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:30.958200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58655 (local address 127.24.103.126:39037)
0807 11:35:30.959119 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:30.959131 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:30.959180 (+ 49us) server_negotiation.cc:407] Connection header received
0807 11:35:30.961338 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:30.961363 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:30.961773 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:30.962144 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:30.963464 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:30.963472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:30.963484 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:30.963503 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:30.963582 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:30.963606 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:30.963799 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:30.963915 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 11:35:30.964223 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":400,"threads_started":1}
I20260807 11:35:30.969208 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:30.972047 18116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57: Bootstrap starting.
I20260807 11:35:30.977315 18116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:30.978717 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:30.975329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46217 (local address 127.24.103.126:39037)
0807 11:35:30.975496 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:30.975500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:30.976119 (+ 619us) server_negotiation.cc:407] Connection header received
0807 11:35:30.976405 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:30.976409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:30.976466 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:30.976559 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:30.977845 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:30.977850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:30.977854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 11:35:30.977871 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:30.977926 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 11:35:30.977945 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:30.978048 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:30.978329 (+ 281us) server_negotiation.cc:299] Negotiation successful
0807 11:35:30.978408 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 11:35:30.979002 18116 log.cc:824] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:30.982260 18116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57: No bootstrap required, opened a new log
I20260807 11:35:30.990677 18116 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce933b1fd02e46b38c705c3698920b57" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39037 } }
I20260807 11:35:30.991176 18116 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 11:35:30.991309 18116 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce933b1fd02e46b38c705c3698920b57, State: Initialized, Role: FOLLOWER
I20260807 11:35:30.992136 18116 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce933b1fd02e46b38c705c3698920b57" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39037 } }
I20260807 11:35:30.992489 18116 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 11:35:30.992662 18116 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 11:35:30.992842 18116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:30.995355 18116 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce933b1fd02e46b38c705c3698920b57" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39037 } }
I20260807 11:35:30.995860 18116 leader_election.cc:302] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce933b1fd02e46b38c705c3698920b57; no voters:
I20260807 11:35:30.996507 18116 leader_election.cc:288] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 11:35:30.996697 18128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:30.997491 18128 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [term 1 LEADER]: Becoming Leader. State: Replica: ce933b1fd02e46b38c705c3698920b57, State: Running, Role: LEADER
I20260807 11:35:30.998062 18128 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce933b1fd02e46b38c705c3698920b57" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39037 } }
I20260807 11:35:30.998895 18116 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:31.000928 18129 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce933b1fd02e46b38c705c3698920b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce933b1fd02e46b38c705c3698920b57" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39037 } } }
I20260807 11:35:31.001971 18129 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:31.001590 18130 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce933b1fd02e46b38c705c3698920b57. Latest consensus state: current_term: 1 leader_uuid: "ce933b1fd02e46b38c705c3698920b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce933b1fd02e46b38c705c3698920b57" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 39037 } } }
I20260807 11:35:31.002529 18130 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:31.004715 18136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:31.010782 18136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:31.020970 18136 catalog_manager.cc:1347] Generated new cluster ID: 79eb0e2587204de08e7fcdc1ca718353
I20260807 11:35:31.021150 18136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:31.048359 18136 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:31.048889 18136 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:31.058000 18136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce933b1fd02e46b38c705c3698920b57: Generated new TSK 0
I20260807 11:35:31.059211 18136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:31.072238 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 18041
W20260807 11:35:31.073539 18111 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58655
W20260807 11:35:31.074376 24989 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 18041: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 11:35:31.074576 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 18041
2026-08-07T11:35:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2217 ms)
[----------] 18 tests from AuthTokenIssuingTest (40331 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:31 dist-test-slave-wl2h krb5kdc[18152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:31 dist-test-slave-wl2h krb5kdc[18152](info): set up 2 sockets
Aug 07 11:35:31 dist-test-slave-wl2h krb5kdc[18152](info): commencing operation
krb5kdc: starting...
W20260807 11:35:32.330037 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.003s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:32 dist-test-slave-wl2h krb5kdc[18152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:35:32.699846 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:37779
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:37779,127.24.103.125:46525,127.24.103.124:44689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:33.062951 18166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:35:33.063414 18166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:33.063489 18166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:33.063601 18166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:33.071403 18166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:33.071959 18166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:33.072290 18166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:33.072391 18166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:33.072522 18166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:33.083341 18166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:37779,127.24.103.125:46525,127.24.103.124:44689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:37779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:33.084810 18166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:33.086169 18166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:33.098606 18172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:33.098641 18175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:33.098678 18173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:33.099911 18166 server_base.cc:1034] running on GCE node
I20260807 11:35:33.100737 18166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:33.102205 18166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:33.103440 18166 hybrid_clock.cc:648] HybridClock initialized: now 1786102533103370 us; error 33 us; skew 500 ppm
I20260807 11:35:33.103824 18166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:33 dist-test-slave-wl2h krb5kdc[18152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102533, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:33.111186 18166 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:35:33.113009 18166 webserver.cc:460] Webserver started at http://127.24.103.126:42923/ using document root <none> and password file <none>
I20260807 11:35:33.113790 18166 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:33.113933 18166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:33.114295 18166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:33.117610 18166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "fdaf26ab04644bd5b283e2671e1ccc87"
format_stamp: "Formatted at 2026-08-07 11:35:33 on dist-test-slave-wl2h"
server_key: "44d950ea6e589b2b4dbac040e35bfeea"
server_key_iv: "586c4b71e4f9d4615c267ce35eda2991"
server_key_version: "encryptionkey@0"
I20260807 11:35:33.118532 18166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "fdaf26ab04644bd5b283e2671e1ccc87"
format_stamp: "Formatted at 2026-08-07 11:35:33 on dist-test-slave-wl2h"
server_key: "44d950ea6e589b2b4dbac040e35bfeea"
server_key_iv: "586c4b71e4f9d4615c267ce35eda2991"
server_key_version: "encryptionkey@0"
I20260807 11:35:33.124238 18166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 11:35:33.128021 18182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:33.129648 18166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 11:35:33.129858 18166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "fdaf26ab04644bd5b283e2671e1ccc87"
format_stamp: "Formatted at 2026-08-07 11:35:33 on dist-test-slave-wl2h"
server_key: "44d950ea6e589b2b4dbac040e35bfeea"
server_key_iv: "586c4b71e4f9d4615c267ce35eda2991"
server_key_version: "encryptionkey@0"
I20260807 11:35:33.130113 18166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:33.150555 18166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:33.154973 18166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:33.155354 18166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:33.176468 18166 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:37779
I20260807 11:35:33.176479 18234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:37779 every 8 connection(s)
I20260807 11:35:33.178027 18166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:33.181886 18235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:33.186582 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 18166
I20260807 11:35:33.186815 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:33.187700 24989 external_mini_cluster.cc:1428] Setting key 6ef37ac04472b1016790ea6ac971d4c0
I20260807 11:35:33.190104 18235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:35:33 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102533, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:35:33.216363 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.194286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:37779 (local address 127.0.0.1:50446)
0807 11:35:33.195296 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.195359 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:33.197741 (+ 2382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:33.198979 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:33.198986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:33.199765 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:33.200148 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:33.200168 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.202546 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.202550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:33.203755 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:33.203768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.204748 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.204754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:33.204962 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:33.205687 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:33.205710 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:33.208331 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:33.212047 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:33.212059 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:33.212072 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:33.212482 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:33.212904 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:33.212908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:33.212912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:33.213062 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:33.213845 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:33.213855 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:33.214081 (+ 226us) client_negotiation.cc:769] Sending connection context
0807 11:35:33.214353 (+ 272us) client_negotiation.cc:240] Negotiation successful
0807 11:35:33.214954 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":147,"thread_start_us":457,"threads_started":1}
I20260807 11:35:33.216325 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.195824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.24.103.126:37779)
0807 11:35:33.196562 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:33.196573 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:33.196630 (+ 57us) server_negotiation.cc:407] Connection header received
0807 11:35:33.198159 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:33.198170 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:33.198497 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:33.198730 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:33.200394 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.202331 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.203910 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.204488 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.204728 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:33.208604 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:33.208612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:33.208625 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:33.208657 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:33.211880 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:33.212626 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:33.212632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:33.212635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:33.212782 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:33.213192 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:33.213195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:33.213197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:33.213706 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:33.213951 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:33.214475 (+ 524us) server_negotiation.cc:299] Negotiation successful
0807 11:35:33.214838 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":506,"thread_start_us":345,"threads_started":1}
I20260807 11:35:33.220237 18235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:35:33.221499 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.221069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43066)
0807 11:35:33.221252 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.221375 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.24.103.125@KRBTEST.COM; defaulting to no policy
W20260807 11:35:33.221846 18186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
I20260807 11:35:33.223644 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.223271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43068)
0807 11:35:33.223435 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.223509 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 11:35:33.223945 18235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:46525: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
I20260807 11:35:33.267628 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:46525
--webserver_interface=127.24.103.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:37779,127.24.103.125:46525,127.24.103.124:44689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:35:33.273350 18235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } attempt: 1
I20260807 11:35:33.274703 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.274170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43082)
0807 11:35:33.274402 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.274550 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 11:35:33.275990 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.275600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43098)
0807 11:35:33.275765 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.275846 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 11:35:33.276214 18235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:46525: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
I20260807 11:35:33.344498 18235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } attempt: 2
I20260807 11:35:33.345803 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.345165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43106)
0807 11:35:33.345469 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.345586 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260807 11:35:33.346993 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.346564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43122)
0807 11:35:33.346746 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.346835 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 11:35:33.347311 18235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:46525: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
I20260807 11:35:33.438608 18235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } attempt: 3
I20260807 11:35:33.440003 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.439379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43130)
0807 11:35:33.439657 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.439796 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260807 11:35:33.441174 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.440735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43134)
0807 11:35:33.440909 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.441005 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 11:35:33.441394 18235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:46525: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
I20260807 11:35:33.584676 18235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } attempt: 4
I20260807 11:35:33.586117 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.585456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43144)
0807 11:35:33.585816 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.585933 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260807 11:35:33.587389 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.586913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43148)
0807 11:35:33.587117 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.587221 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 11:35:33.587617 18235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:46525: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46525: connect: Connection refused (error 111)
W20260807 11:35:33.626591 18242 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:35:33.626945 18242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:33.627045 18242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:33.627195 18242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:33.635100 18242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:33.635206 18242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:33.635265 18242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:33.635315 18242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:33.635378 18242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:33.645634 18242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:37779,127.24.103.125:46525,127.24.103.124:44689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:46525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.103.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:33.646966 18242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:33.648522 18242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:33.661485 18249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:33.661525 18253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:33.661578 18250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:33.662664 18242 server_base.cc:1034] running on GCE node
I20260807 11:35:33.663643 18242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:33.665099 18242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:33.666357 18242 hybrid_clock.cc:648] HybridClock initialized: now 1786102533666286 us; error 37 us; skew 500 ppm
I20260807 11:35:33.666673 18242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:33 dist-test-slave-wl2h krb5kdc[18152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102533, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:33.670288 18242 init.cc:377] Logged in from keytab as kudu/127.24.103.125@KRBTEST.COM (short username kudu)
I20260807 11:35:33.671973 18242 webserver.cc:460] Webserver started at http://127.24.103.125:42863/ using document root <none> and password file <none>
I20260807 11:35:33.672737 18242 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:33.672855 18242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:33.673208 18242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:33.676375 18242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/data/instance:
uuid: "d1647e74a0cd44f191f56687cf2915c0"
format_stamp: "Formatted at 2026-08-07 11:35:33 on dist-test-slave-wl2h"
server_key: "21b4af028cd2495ea3d6496142092dea"
server_key_iv: "97d97e1fd2d59663e5158e553a32a6d9"
server_key_version: "encryptionkey@0"
I20260807 11:35:33.677225 18242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/wal/instance:
uuid: "d1647e74a0cd44f191f56687cf2915c0"
format_stamp: "Formatted at 2026-08-07 11:35:33 on dist-test-slave-wl2h"
server_key: "21b4af028cd2495ea3d6496142092dea"
server_key_iv: "97d97e1fd2d59663e5158e553a32a6d9"
server_key_version: "encryptionkey@0"
I20260807 11:35:33.682612 18242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 11:35:33.686214 18261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:33.687680 18242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:35:33.687868 18242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/wal
uuid: "d1647e74a0cd44f191f56687cf2915c0"
format_stamp: "Formatted at 2026-08-07 11:35:33 on dist-test-slave-wl2h"
server_key: "21b4af028cd2495ea3d6496142092dea"
server_key_iv: "97d97e1fd2d59663e5158e553a32a6d9"
server_key_version: "encryptionkey@0"
I20260807 11:35:33.688105 18242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:33.704484 18242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:33.709174 18242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:33.709501 18242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:33.729928 18242 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.125:46525
I20260807 11:35:33.729952 18320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.125:46525 every 8 connection(s)
I20260807 11:35:33.731462 18242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
I20260807 11:35:33.734895 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 18242
I20260807 11:35:33.735180 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-1/wal/instance
I20260807 11:35:33.735674 24989 external_mini_cluster.cc:1428] Setting key 0b9e8528a6f8637489fc634b682307c0
I20260807 11:35:33.735826 18321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:33.742666 18321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:35:33 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102533, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:35:33.765338 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.745459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.24.103.126:37779)
0807 11:35:33.745995 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:33.745999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:33.747128 (+ 1129us) server_negotiation.cc:407] Connection header received
0807 11:35:33.749140 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:33.749146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:33.749243 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:33.749370 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:33.751168 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.751882 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.753431 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.753865 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.754004 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:33.757608 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:33.757616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:33.757624 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:33.757688 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:33.761603 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:33.762431 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:33.762440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:33.762446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:33.762675 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:33.763177 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:33.763185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:33.763191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:33.763638 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:33.763800 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:33.764561 (+ 761us) server_negotiation.cc:299] Negotiation successful
0807 11:35:33.764835 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":298,"threads_started":1}
I20260807 11:35:33.765973 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.746064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:37779 (local address 127.0.0.1:50460)
0807 11:35:33.746904 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.746944 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:33.748857 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:33.749611 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:33.749619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:33.750141 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:33.750904 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:33.750923 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.752071 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.752075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:33.753265 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:33.753281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.754094 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.754101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:33.754351 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:33.755140 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:33.755162 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:33.757349 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:33.761762 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:33.761775 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:33.761791 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:33.762280 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:33.762809 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:33.762812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:33.762815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:33.763000 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:33.763768 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:33.763775 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:33.764120 (+ 345us) client_negotiation.cc:769] Sending connection context
0807 11:35:33.764385 (+ 265us) client_negotiation.cc:240] Negotiation successful
0807 11:35:33.764779 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":371,"threads_started":1}
I20260807 11:35:33.767817 18321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.103.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.124@KRBTEST.COM" created.
Aug 07 11:35:33 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102533, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260807 11:35:33.784961 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.768781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43160)
0807 11:35:33.768991 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.769010 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:33.769125 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:33.770328 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:33.770334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:33.770743 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:33.770889 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:33.770897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.772811 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.772815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:33.773797 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:33.773809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.774642 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.774647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:33.774874 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:33.775581 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:33.775598 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:33.777580 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:33.781902 (+ 4322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:33.781908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:33.781913 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:33.782334 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:33.782779 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:33.782785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:33.782789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:33.782938 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:33.783878 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:33.783883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:33.784041 (+ 158us) client_negotiation.cc:769] Sending connection context
0807 11:35:33.784185 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 11:35:33.784502 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":5}
I20260807 11:35:33.785006 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.768871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.24.103.125:46525)
0807 11:35:33.769741 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:33.769752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:33.769789 (+ 37us) server_negotiation.cc:407] Connection header received
0807 11:35:33.769852 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:33.769861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:33.770128 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:33.770300 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:33.771225 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.772648 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.773934 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.774482 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.774628 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:33.777802 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:33.777814 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:33.777828 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:33.777858 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:33.781710 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:33.782495 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:33.782506 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:33.782510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:33.782647 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:33.783099 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:33.783104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:33.783107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:33.783719 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:33.783967 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:33.784392 (+ 425us) server_negotiation.cc:299] Negotiation successful
0807 11:35:33.784624 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":643,"thread_start_us":490,"threads_started":1}
I20260807 11:35:33.787719 18321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:35:33.789081 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.788473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37886)
0807 11:35:33.788709 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.788857 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260807 11:35:33.789379 18265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
I20260807 11:35:33.791373 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.790867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37892)
0807 11:35:33.791168 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.791255 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260807 11:35:33.791617 18321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44689: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
I20260807 11:35:33.818573 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:44689
--webserver_interface=127.24.103.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:37779,127.24.103.125:46525,127.24.103.124:44689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:35:33.840989 18321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } attempt: 1
I20260807 11:35:33.842326 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.841805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37908)
0807 11:35:33.842026 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.842158 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 11:35:33.843459 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.843070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37914)
0807 11:35:33.843259 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.843345 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 11:35:33.843667 18321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44689: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
I20260807 11:35:33.886953 18235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } attempt: 5
Aug 07 11:35:33 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102533, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260807 11:35:33.902837 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.888120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43172 (local address 127.24.103.125:46525)
0807 11:35:33.888360 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:33.888366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:33.888389 (+ 23us) server_negotiation.cc:407] Connection header received
0807 11:35:33.888457 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:33.888462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:33.888565 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:33.888683 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:33.889525 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.890655 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.892529 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.893371 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.893478 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:33.897376 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:33.897381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:33.897384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:33.897419 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:33.900134 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:33.900828 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:33.900834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:33.900837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:33.900974 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:33.901370 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:33.901375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:33.901379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:33.901668 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:33.901841 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:33.902220 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 11:35:33.902461 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 11:35:33.902841 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.887843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43172)
0807 11:35:33.888081 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.888106 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:33.888254 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:33.888769 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:33.888773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:33.889174 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:33.889352 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:33.889360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.890851 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.890856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:33.892321 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:33.892339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:33.893537 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:33.893543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:33.893781 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:33.894553 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:33.894578 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:33.897092 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:33.900296 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:33.900302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:33.900307 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:33.900700 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:33.901094 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:33.901100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:33.901103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:33.901255 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:33.901800 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:33.901805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:33.901971 (+ 166us) client_negotiation.cc:769] Sending connection context
0807 11:35:33.902117 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 11:35:33.902462 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 11:35:33.904162 18235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:35:33.905212 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.904825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37926)
0807 11:35:33.904991 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.905071 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 11:35:33.906090 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.905781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37938)
0807 11:35:33.905937 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.906001 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 11:35:33.906313 18235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44689: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
I20260807 11:35:33.911908 18321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } attempt: 2
I20260807 11:35:33.913192 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.912601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37946)
0807 11:35:33.912851 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.912976 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 11:35:33.914486 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.914028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37958)
0807 11:35:33.914212 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.914313 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 11:35:33.914742 18321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44689: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
I20260807 11:35:33.957571 18235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } attempt: 1
I20260807 11:35:33.958971 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.958423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37970)
0807 11:35:33.958686 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.958786 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260807 11:35:33.960263 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:33.959890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37974)
0807 11:35:33.960053 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:33.960141 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 11:35:33.960518 18235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44689: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
I20260807 11:35:34.006013 18321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } attempt: 3
I20260807 11:35:34.007396 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.006783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37984)
0807 11:35:34.007061 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.007191 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260807 11:35:34.008649 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.008168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37986)
0807 11:35:34.008406 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.008502 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260807 11:35:34.008946 18321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44689: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
I20260807 11:35:34.028782 18235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } attempt: 2
I20260807 11:35:34.030313 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.029551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37990)
0807 11:35:34.029902 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.030048 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260807 11:35:34.031718 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.031225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37996)
0807 11:35:34.031426 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.031535 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 11:35:34.032001 18235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44689: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
I20260807 11:35:34.138286 18235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } attempt: 3
I20260807 11:35:34.139617 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.138989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:37998)
0807 11:35:34.139298 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.139409 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260807 11:35:34.140805 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.140372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:38012)
0807 11:35:34.140550 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.140649 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260807 11:35:34.141077 18235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44689: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
I20260807 11:35:34.152280 18321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } attempt: 4
I20260807 11:35:34.153594 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.152944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:38020)
0807 11:35:34.153234 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.153370 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 11:35:34.154913 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.154456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:38032)
0807 11:35:34.154639 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.154743 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 11:35:34.155260 18321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:44689: Network error: Client connection negotiation failed: client connection to 127.24.103.124:44689: connect: Connection refused (error 111)
W20260807 11:35:34.180156 18330 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:35:34.180527 18330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:34.180660 18330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:34.180799 18330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:34.188711 18330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:34.188843 18330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:34.188972 18330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:34.189065 18330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:34.189177 18330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:34.200074 18330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:37779,127.24.103.125:46525,127.24.103.124:44689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:44689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.103.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:34.202243 18330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:34.203913 18330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:34.216531 18339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:34.216620 18335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:34.216595 18337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:34.217777 18330 server_base.cc:1034] running on GCE node
I20260807 11:35:34.218787 18330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:34.220476 18330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:34.221801 18330 hybrid_clock.cc:648] HybridClock initialized: now 1786102534221732 us; error 66 us; skew 500 ppm
I20260807 11:35:34.222186 18330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:34 dist-test-slave-wl2h krb5kdc[18152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102534, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:34.234118 18330 init.cc:377] Logged in from keytab as kudu/127.24.103.124@KRBTEST.COM (short username kudu)
I20260807 11:35:34.235962 18330 webserver.cc:460] Webserver started at http://127.24.103.124:33543/ using document root <none> and password file <none>
I20260807 11:35:34.236712 18330 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:34.236853 18330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:34.237196 18330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:34.240332 18330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/data/instance:
uuid: "65c2bd7bdd2a4651801730fb3bc2fff6"
format_stamp: "Formatted at 2026-08-07 11:35:34 on dist-test-slave-wl2h"
server_key: "3996f5be02b4aade5ef2913dd7f6063a"
server_key_iv: "6935c2ee837173c06cf69614f381e15c"
server_key_version: "encryptionkey@0"
I20260807 11:35:34.241137 18330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/wal/instance:
uuid: "65c2bd7bdd2a4651801730fb3bc2fff6"
format_stamp: "Formatted at 2026-08-07 11:35:34 on dist-test-slave-wl2h"
server_key: "3996f5be02b4aade5ef2913dd7f6063a"
server_key_iv: "6935c2ee837173c06cf69614f381e15c"
server_key_version: "encryptionkey@0"
I20260807 11:35:34.246533 18330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 11:35:34.250547 18346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:34.251942 18330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 11:35:34.252144 18330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/wal
uuid: "65c2bd7bdd2a4651801730fb3bc2fff6"
format_stamp: "Formatted at 2026-08-07 11:35:34 on dist-test-slave-wl2h"
server_key: "3996f5be02b4aade5ef2913dd7f6063a"
server_key_iv: "6935c2ee837173c06cf69614f381e15c"
server_key_version: "encryptionkey@0"
I20260807 11:35:34.252384 18330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:34.266516 18330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:34.271333 18330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:34.271641 18330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:34.292444 18330 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.124:44689
I20260807 11:35:34.292487 18410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.124:44689 every 8 connection(s)
I20260807 11:35:34.293969 18330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
I20260807 11:35:34.295451 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 18330
I20260807 11:35:34.295747 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786102153172159-24989-0/minicluster-data/master-2/wal/instance
I20260807 11:35:34.296284 24989 external_mini_cluster.cc:1428] Setting key 13bcdf94289e80f474d8bb17fddc2c10
I20260807 11:35:34.299226 18411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:34.307837 18411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:35:34 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260807 11:35:34.318320 18235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } attempt: 4
Aug 07 11:35:34 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102534, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:35:34.323066 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.298261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.24.103.124:44689)
0807 11:35:34.299180 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.299194 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.299242 (+ 48us) server_negotiation.cc:407] Connection header received
0807 11:35:34.301017 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.301042 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.301444 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.301810 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.303785 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.305325 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.307187 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.307866 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.308008 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.313227 (+ 5219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:34.313235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:34.313244 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:34.313278 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:34.316938 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.317848 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.317854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.317861 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.318091 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.318591 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.318596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.318599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.319420 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:34.320611 (+ 1191us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.320855 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.321436 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":412,"threads_started":1}
Aug 07 11:35:34 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102533, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260807 11:35:34.332620 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.311159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.24.103.126:37779)
0807 11:35:34.311738 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.311746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.312245 (+ 499us) server_negotiation.cc:407] Connection header received
0807 11:35:34.312479 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.312484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.312549 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.312669 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.313849 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.314628 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.316730 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.317238 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.317388 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.321387 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:34.321394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:34.321398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:34.321447 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:34.328995 (+ 7548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.329856 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.329862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.329865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.330002 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.330681 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.330686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.330690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.331045 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:34.331182 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.332003 (+ 821us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.332244 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":285,"threads_started":1}
I20260807 11:35:34.333763 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.311156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:37779 (local address 127.0.0.1:50468)
0807 11:35:34.312074 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.312148 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:34.312319 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:34.312738 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:34.312746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:34.313391 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:34.313681 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:34.313689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.314803 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.314812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:34.316535 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:34.316555 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.317424 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.317428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:34.317553 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.318126 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:34.318160 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:34.321169 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:34.329149 (+ 7980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:34.329159 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:34.329166 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:34.329713 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:34.330134 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:34.330139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:34.330143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:34.330329 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:34.331260 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:34.331268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:34.331486 (+ 218us) client_negotiation.cc:769] Sending connection context
0807 11:35:34.331656 (+ 170us) client_negotiation.cc:240] Negotiation successful
0807 11:35:34.333357 (+ 1701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":534,"threads_started":1}
I20260807 11:35:34.335620 18411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:35:34.337013 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.318956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:38044)
0807 11:35:34.319279 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.319302 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:34.319423 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:34.320403 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:34.320410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:34.320931 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:34.321149 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:34.321161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.323095 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.323102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:34.324705 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:34.324722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.325753 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.325759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:34.325948 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.326622 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:34.326647 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:34.329458 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:34.334000 (+ 4542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:34.334004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:34.334007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:34.334584 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:34.335133 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:34.335138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:34.335144 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:34.335333 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:34.335979 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:34.335986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:34.336161 (+ 175us) client_negotiation.cc:769] Sending connection context
0807 11:35:34.336322 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 11:35:34.336601 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260807 11:35:34.338176 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.319237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.24.103.124:44689)
0807 11:35:34.320020 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.320026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.320048 (+ 22us) server_negotiation.cc:407] Connection header received
0807 11:35:34.320093 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.320098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.320165 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.321307 (+ 1142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.321586 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.322896 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.324910 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.325550 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.326255 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.329710 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:34.329716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:34.329721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:34.329757 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:34.333113 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.334791 (+ 1678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.334796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.334799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.334977 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.335487 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.335491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.335494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.335835 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:34.337379 (+ 1544us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.337492 (+ 113us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.337727 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":445,"threads_started":1}
Aug 07 11:35:34 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102534, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260807 11:35:34.345602 18235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87: Bootstrap starting.
Aug 07 11:35:34 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:35:34.350595 18235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:34.350889 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.336684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.24.103.125:46525)
0807 11:35:34.336877 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.336882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.337140 (+ 258us) server_negotiation.cc:407] Connection header received
0807 11:35:34.337398 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.337404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.337485 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.337556 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.338856 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.339765 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.340956 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.341370 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.341514 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.345003 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:34.345012 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:34.345020 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:34.345249 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:34.348107 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.348783 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.348789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.348793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.348938 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.349340 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.349346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.349349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.349766 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:34.349930 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.350296 (+ 366us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.350508 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 11:35:34.350904 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.336322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46525 (local address 127.0.0.1:43176)
0807 11:35:34.337047 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.337067 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:34.337239 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:34.337783 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:34.337787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:34.338387 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:34.338660 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:34.338671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.339953 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.339957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:34.340807 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:34.340817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.341539 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.341545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:34.341735 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.342325 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:34.342350 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:34.344769 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:34.348255 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:34.348262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:34.348266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:34.348658 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:34.349056 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:34.349061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:34.349064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:34.349226 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:34.349888 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:34.349893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:34.350047 (+ 154us) client_negotiation.cc:769] Sending connection context
0807 11:35:34.350189 (+ 142us) client_negotiation.cc:240] Negotiation successful
0807 11:35:34.350481 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605}
I20260807 11:35:34.352172 18411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:35:34.352231 18235 log.cc:824] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:34.356881 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.334057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.24.103.126:37779)
0807 11:35:34.334529 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.334537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.334557 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:35:34.334614 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.334623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.334732 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.336679 (+ 1947us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.336828 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.338698 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.340739 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.341489 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.342238 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.347402 (+ 5164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:34.347411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:34.347417 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:34.347453 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:34.351573 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.352580 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.352586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.352590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.352747 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.353315 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.353323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.353330 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.353864 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:34.355836 (+ 1972us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.356033 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.356355 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
Aug 07 11:35:34 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102534, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260807 11:35:34.363490 18235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87: No bootstrap required, opened a new log
Aug 07 11:35:34 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260807 11:35:34.371889 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.352841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:38056)
0807 11:35:34.353135 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.353153 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:34.353267 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:34.353937 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:34.353942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:34.354423 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:34.354613 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:34.354625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.356003 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.356008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:34.357754 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:34.357772 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.358831 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.358838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:34.359150 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.359977 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:34.359999 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:34.364563 (+ 4564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:34.368945 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:34.368953 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:34.368958 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:34.369518 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:34.369974 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:34.369979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:34.369984 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:34.370138 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:34.370784 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:34.370792 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:34.370957 (+ 165us) client_negotiation.cc:769] Sending connection context
0807 11:35:34.371131 (+ 174us) client_negotiation.cc:240] Negotiation successful
0807 11:35:34.371459 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":2}
I20260807 11:35:34.373564 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.352878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38056 (local address 127.24.103.124:44689)
0807 11:35:34.353476 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.353483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.353500 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:35:34.353553 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.353559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.353659 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.353771 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.354779 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.355842 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.357951 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.358642 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.358825 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.364811 (+ 5986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:34.364818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:34.364822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:34.364858 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:34.368788 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.369677 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.369684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.369687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.369852 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.370271 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.370275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.370278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.370630 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:34.372670 (+ 2040us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.372848 (+ 178us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.373129 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":479}
I20260807 11:35:34.375465 18235 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } }
I20260807 11:35:34.375967 18235 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdaf26ab04644bd5b283e2671e1ccc87, State: Initialized, Role: FOLLOWER
I20260807 11:35:34.376534 18235 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } }
I20260807 11:35:34.379904 18411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6: Bootstrap starting.
I20260807 11:35:34.381242 18422 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } } }
I20260807 11:35:34.381711 18422 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:34.382503 18235 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:34.384406 18411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:34.384971 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.359231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.24.103.125:46525)
0807 11:35:34.359446 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.359453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.359881 (+ 428us) server_negotiation.cc:407] Connection header received
0807 11:35:34.359945 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.359950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.360034 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.360140 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.362301 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.363651 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.365481 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.366145 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.366279 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.371942 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:34.371949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:34.371954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:34.371995 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:34.375733 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.376727 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.376734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.376738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.376915 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.377503 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.377512 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.377517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.377972 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:34.378321 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.384028 (+ 5707us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.384441 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 11:35:34.386174 18411 log.cc:824] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:34.394168 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:34.398329 18209 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fdaf26ab04644bd5b283e2671e1ccc87"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50472
I20260807 11:35:34.398867 18209 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:35:34.399108 18209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:34.401556 18411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6: No bootstrap required, opened a new log
I20260807 11:35:34.402117 18209 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } }
I20260807 11:35:34.403548 18209 leader_election.cc:288] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [CANDIDATE]: Term 1 election: Requested vote from peers d1647e74a0cd44f191f56687cf2915c0 (127.24.103.125:46525), 65c2bd7bdd2a4651801730fb3bc2fff6 (127.24.103.124:44689)
W20260807 11:35:34.408411 18188 tablet.cc:2369] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:35:34.418874 18411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } }
I20260807 11:35:34.420018 18411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c2bd7bdd2a4651801730fb3bc2fff6, State: Initialized, Role: FOLLOWER
I20260807 11:35:34.420688 18411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } }
I20260807 11:35:34.426291 18411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:34.426291 18380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6"
I20260807 11:35:34.426874 18380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:34.430713 18380 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdaf26ab04644bd5b283e2671e1ccc87 in term 1.
I20260807 11:35:34.431540 18186 leader_election.cc:302] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c2bd7bdd2a4651801730fb3bc2fff6, fdaf26ab04644bd5b283e2671e1ccc87; no voters:
I20260807 11:35:34.432189 18422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:34.432056 18440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } } }
I20260807 11:35:34.432682 18440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:34.434033 18422 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [term 1 LEADER]: Becoming Leader. State: Replica: fdaf26ab04644bd5b283e2671e1ccc87, State: Running, Role: LEADER
I20260807 11:35:34.434723 18422 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } }
I20260807 11:35:34.440969 18445 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdaf26ab04644bd5b283e2671e1ccc87. Latest consensus state: current_term: 1 leader_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } } }
I20260807 11:35:34.441375 18445 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:34.442301 18448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:34.451202 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.408916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.24.103.125:46525)
0807 11:35:34.409125 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.409132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.412633 (+ 3501us) server_negotiation.cc:407] Connection header received
0807 11:35:34.413002 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.413011 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.413109 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.413236 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.423022 (+ 9786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.424521 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.435038 (+ 10517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.435748 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.435848 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.441250 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:34.441258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:34.441263 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:34.441313 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:34.445489 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.446295 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.446303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.446310 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.446634 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.447590 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.447597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.447603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.448479 (+ 876us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:34.449159 (+ 680us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.449437 (+ 278us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.449966 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 11:35:34.453478 18448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:34.454459 18321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } attempt: 5
I20260807 11:35:34.457844 18453 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 11:35:34.461679 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.409359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.24.103.126:37779)
0807 11:35:34.409629 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.409638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.422256 (+ 12618us) server_negotiation.cc:407] Connection header received
0807 11:35:34.425003 (+ 2747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.425013 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.425136 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.425225 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.435004 (+ 9779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.436103 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.438184 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.438943 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.439086 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.450105 (+ 11019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:34.450112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:34.450116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:34.450158 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:34.453996 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.455503 (+ 1507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.455519 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.455525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.455937 (+ 412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.457059 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.457096 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.457103 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.458034 (+ 931us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:34.458596 (+ 562us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.460049 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.460654 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260807 11:35:34.468148 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.430028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.24.103.124:44689)
0807 11:35:34.430187 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.430192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.432533 (+ 2341us) server_negotiation.cc:407] Connection header received
0807 11:35:34.433547 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.433554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.433647 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.433728 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.445142 (+ 11414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.446472 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.456050 (+ 9578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.456882 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.456984 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.460463 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:34.460469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:34.460473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:34.460512 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:34.464142 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.465041 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.465050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.465056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.465289 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.465917 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.465925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.465931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.466391 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:34.466818 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.467277 (+ 459us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.467603 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 07 11:35:34 dist-test-slave-wl2h krb5kdc[18152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102533, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260807 11:35:34.462612 18166 master_runner.cc:186] Error getting master registration for 127.24.103.125:46525: OK, instance_id { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" instance_seqno: 1786102533724111 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:35:34.482755 18166 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 11:35:34.487824 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.455034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:44689 (local address 127.0.0.1:38066)
0807 11:35:34.455473 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:34.455492 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:34.455586 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:34.456526 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:34.456555 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:34.457077 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:34.457296 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:34.457311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.468315 (+ 11004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.468321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:34.469812 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:34.469833 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.471083 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.471090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:34.471343 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.472160 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:34.472215 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:34.476052 (+ 3837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:34.481734 (+ 5682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:34.481763 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:34.481767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:34.482618 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:34.483466 (+ 848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:34.483499 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:34.483504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:34.483839 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:34.485072 (+ 1233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:34.485099 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:34.485378 (+ 279us) client_negotiation.cc:769] Sending connection context
0807 11:35:34.485599 (+ 221us) client_negotiation.cc:240] Negotiation successful
0807 11:35:34.486300 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299}
I20260807 11:35:34.489159 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.455813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.24.103.124:44689)
0807 11:35:34.456028 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.456033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.456048 (+ 15us) server_negotiation.cc:407] Connection header received
0807 11:35:34.456090 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.456144 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.456237 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.456351 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.457800 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.458957 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.470038 (+ 11081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.470720 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.470842 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.476327 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:34.476333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:34.476337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:34.476374 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:34.481484 (+ 5110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.482897 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.482902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.482905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.483255 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:34.484104 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:34.484109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:34.484112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:34.484856 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:34.486258 (+ 1402us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.486769 (+ 511us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.487890 (+ 1121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 11:35:34.495599 18321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0: Bootstrap starting.
I20260807 11:35:34.501652 18321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:34.503968 18321 log.cc:824] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:34.512305 18380 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [term 1 FOLLOWER]: Refusing update from remote peer fdaf26ab04644bd5b283e2671e1ccc87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:35:34.514612 18445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:35:34.518658 18321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0: No bootstrap required, opened a new log
W20260807 11:35:34.523900 18354 tablet.cc:2369] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:35:34.530351 18321 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } }
I20260807 11:35:34.530923 18321 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1647e74a0cd44f191f56687cf2915c0, State: Initialized, Role: FOLLOWER
I20260807 11:35:34.534277 18321 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } }
I20260807 11:35:34.540213 18455 mvcc.cc:205] Tried to move back new op lower bound from 7315875981349634048 to 7315875981053317120. Current Snapshot: MvccSnapshot[applied={T|T < 7315875981349634048}]
I20260807 11:35:34.541394 18466 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } } }
I20260807 11:35:34.543962 18292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d1647e74a0cd44f191f56687cf2915c0"
I20260807 11:35:34.544384 18291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:34.543932 18321 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:34.546133 18466 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:34.553805 18468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } } }
I20260807 11:35:34.554366 18468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:34.553555 18291 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [term 1 FOLLOWER]: Refusing update from remote peer fdaf26ab04644bd5b283e2671e1ccc87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:35:34.554924 18440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdaf26ab04644bd5b283e2671e1ccc87. Latest consensus state: current_term: 1 leader_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } } }
I20260807 11:35:34.556584 18440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:34.560482 18292 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdaf26ab04644bd5b283e2671e1ccc87 in term 1.
I20260807 11:35:34.562547 18448 catalog_manager.cc:1347] Generated new cluster ID: 9f351265cd394930a70df547ee3bc215
I20260807 11:35:34.562662 18448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:34.563187 18445 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } } }
I20260807 11:35:34.563472 18445 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [sys.catalog]: This master's current role is: LEADER
W20260807 11:35:34.569600 18474 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:35:34.569741 18474 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:35:34.578527 18422 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:35:34.569249 18330 master_runner.cc:186] Error getting master registration for 127.24.103.125:46525: OK, instance_id { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" instance_seqno: 1786102533724111 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:35:34.590003 18330 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 11:35:34.591696 18448 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:34.598930 18448 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:34.628492 18448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87: Generated new TSK 0
I20260807 11:35:34.629987 18448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 11:35:34.644235 18466 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdaf26ab04644bd5b283e2671e1ccc87. Latest consensus state: current_term: 1 leader_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } } }
I20260807 11:35:34.644797 18466 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:34.645277 18485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } } }
I20260807 11:35:34.645843 18485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:34.647778 18422 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdaf26ab04644bd5b283e2671e1ccc87" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 37779 } } peers { permanent_uuid: "d1647e74a0cd44f191f56687cf2915c0" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46525 } } peers { permanent_uuid: "65c2bd7bdd2a4651801730fb3bc2fff6" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 44689 } } }
I20260807 11:35:34.648176 18422 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdaf26ab04644bd5b283e2671e1ccc87 [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:34.709580 18490 catalog_manager.cc:1259] Loaded cluster ID: 9f351265cd394930a70df547ee3bc215
I20260807 11:35:34.709703 18490 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0: loading cluster ID for follower catalog manager: success
I20260807 11:35:34.713681 18490 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0: acquiring CA information for follower catalog manager: success
I20260807 11:35:34.716940 18490 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1647e74a0cd44f191f56687cf2915c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 11:35:34.779670 24989 mini_kdc.cc:348] Time spent kdestroy: real 0.124s user 0.001s sys 0.030s
W20260807 11:35:34.847602 18418 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:34.837339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50488 (local address 127.24.103.126:37779)
0807 11:35:34.837776 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.837783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.844267 (+ 6484us) server_negotiation.cc:407] Connection header received
0807 11:35:34.845955 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.845962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.846055 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.846135 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.847347 (+ 1212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50488: BlockingRecv error: recv got EOF from 127.0.0.1:50488 (error 108)
Metrics: {"server-negotiator.queue_time_us":274}
W20260807 11:35:34.849802 18327 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:34.837145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.24.103.125:46525)
0807 11:35:34.837564 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.837571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.843743 (+ 6172us) server_negotiation.cc:407] Connection header received
0807 11:35:34.848179 (+ 4436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.848185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.848296 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.848370 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.849538 (+ 1168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43216: BlockingRecv error: recv got EOF from 127.0.0.1:43216 (error 108)
Metrics: {"server-negotiator.queue_time_us":261}
W20260807 11:35:34.853555 18417 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:34.844903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.24.103.124:44689)
0807 11:35:34.845115 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.845122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.851358 (+ 6236us) server_negotiation.cc:407] Connection header received
0807 11:35:34.852135 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.852142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.852232 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.852305 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.853294 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38072: BlockingRecv error: recv got EOF from 127.0.0.1:38072 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:35:34.905452 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.885753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.24.103.125:46525)
0807 11:35:34.885958 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.885965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.897180 (+ 11215us) server_negotiation.cc:407] Connection header received
0807 11:35:34.897573 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.897580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.897801 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.897946 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:34.898750 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.899965 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.902200 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.902911 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.903040 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:34.903832 (+ 792us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:34.904581 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:34.904704 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:34.904966 (+ 262us) server_negotiation.cc:299] Negotiation successful
0807 11:35:34.905089 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 11:35:34.949182 18417 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:34.939574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.24.103.124:44689)
0807 11:35:34.939784 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.939790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.947106 (+ 7316us) server_negotiation.cc:407] Connection header received
0807 11:35:34.947475 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.947482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.947619 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.947704 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:34.948890 (+ 1186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38086: BlockingRecv error: recv got EOF from 127.0.0.1:38086 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 11:35:35.002324 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:34.987465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.24.103.125:46525)
0807 11:35:34.987668 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:34.987685 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:34.995102 (+ 7417us) server_negotiation.cc:407] Connection header received
0807 11:35:34.995660 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:34.995666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:34.995777 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:34.995848 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:34.996566 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:34.997795 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:34.999947 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.000601 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.000723 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.001379 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.001557 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.001628 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.001880 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.001977 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260807 11:35:35.047395 18417 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:35.037442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.24.103.124:44689)
0807 11:35:35.037667 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.037675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.044771 (+ 7096us) server_negotiation.cc:407] Connection header received
0807 11:35:35.045431 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.045439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.045566 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.045655 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:35.047098 (+ 1443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38088: BlockingRecv error: recv got EOF from 127.0.0.1:38088 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 11:35:35.100781 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.086367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.24.103.125:46525)
0807 11:35:35.086573 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.086580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.093627 (+ 7047us) server_negotiation.cc:407] Connection header received
0807 11:35:35.094160 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.094167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.094278 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.094356 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.095134 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.096386 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.098414 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.099146 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.099310 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.099838 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.100012 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.100121 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.100324 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.100425 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 11:35:35.115574 18417 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:35.113161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.24.103.124:44689)
0807 11:35:35.113375 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.113381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.113950 (+ 569us) server_negotiation.cc:407] Connection header received
0807 11:35:35.114144 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.114149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.114290 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.114407 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:35.115321 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38102: BlockingRecv error: recv got EOF from 127.0.0.1:38102 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 11:35:35.136279 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.128643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.24.103.125:46525)
0807 11:35:35.128845 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.128852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.129439 (+ 587us) server_negotiation.cc:407] Connection header received
0807 11:35:35.129648 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.129653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.129766 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.129878 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.130384 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.131686 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.133759 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.134555 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.134738 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.135450 (+ 712us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.135587 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.135681 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.135885 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.135971 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 11:35:35.151314 18417 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:35.143782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.24.103.124:44689)
0807 11:35:35.144058 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.144069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.147640 (+ 3571us) server_negotiation.cc:407] Connection header received
0807 11:35:35.147865 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.147874 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.147998 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.148118 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:35.150984 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59640: BlockingRecv error: recv got EOF from 127.0.0.1:59640 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
I20260807 11:35:35.169159 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.161720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.24.103.125:46525)
0807 11:35:35.161919 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.161925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.162504 (+ 579us) server_negotiation.cc:407] Connection header received
0807 11:35:35.162738 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.162745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.162851 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.162965 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.163560 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.164820 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.166832 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.167586 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.167731 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.168255 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.168444 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.168540 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.168735 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.168827 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 11:35:35.195876 18417 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:35.176650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.24.103.124:44689)
0807 11:35:35.176845 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.176851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.192679 (+ 15828us) server_negotiation.cc:407] Connection header received
0807 11:35:35.192817 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.192824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.192933 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.193039 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:35.195620 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59642: BlockingRecv error: recv got EOF from 127.0.0.1:59642 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 11:35:35.224581 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.216588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.24.103.125:46525)
0807 11:35:35.216783 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.216790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.217344 (+ 554us) server_negotiation.cc:407] Connection header received
0807 11:35:35.218600 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.218607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.218703 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.218826 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.219359 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.220618 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.222510 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.223225 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.223387 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.223790 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.223940 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.224052 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.224164 (+ 112us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.224265 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260807 11:35:35.233191 18417 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:35.230929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.24.103.124:44689)
0807 11:35:35.231131 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.231135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.231657 (+ 522us) server_negotiation.cc:407] Connection header received
0807 11:35:35.231887 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.231891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.231995 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.232121 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:35.232919 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59650: BlockingRecv error: recv got EOF from 127.0.0.1:59650 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 11:35:35.276163 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.269414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.24.103.125:46525)
0807 11:35:35.269677 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.269684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.270001 (+ 317us) server_negotiation.cc:407] Connection header received
0807 11:35:35.270213 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.270220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.270379 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.270502 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.270994 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.272249 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.273765 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.274545 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.274675 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.275107 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.275252 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.275346 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.275723 (+ 377us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.275821 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260807 11:35:35.285010 18417 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:35.282704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.24.103.124:44689)
0807 11:35:35.282911 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.282918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.283426 (+ 508us) server_negotiation.cc:407] Connection header received
0807 11:35:35.283680 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.283686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.283782 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.283897 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:35.284759 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59652: BlockingRecv error: recv got EOF from 127.0.0.1:59652 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 11:35:35.357726 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.352621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35888 (local address 127.24.103.125:46525)
0807 11:35:35.352834 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.352839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.353125 (+ 286us) server_negotiation.cc:407] Connection header received
0807 11:35:35.353345 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.353349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.353422 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.353513 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.353902 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.354674 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.356056 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.356551 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.356648 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.357092 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.357208 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.357282 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.357429 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.357495 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260807 11:35:35.365600 18417 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:35.363546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.24.103.124:44689)
0807 11:35:35.363833 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.363840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.364183 (+ 343us) server_negotiation.cc:407] Connection header received
0807 11:35:35.364418 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.364422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.364497 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.364612 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:35.365406 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59662: BlockingRecv error: recv got EOF from 127.0.0.1:59662 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260807 11:35:35.503901 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.497843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.24.103.125:46525)
0807 11:35:35.498095 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.498103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.498582 (+ 479us) server_negotiation.cc:407] Connection header received
0807 11:35:35.498850 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.498855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.498928 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.499055 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.499539 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.500261 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.501644 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.502132 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.502289 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.502747 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.502961 (+ 214us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.503192 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.503480 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.503576 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260807 11:35:35.512720 18417 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:35.510649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.24.103.124:44689)
0807 11:35:35.510887 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.510893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.511300 (+ 407us) server_negotiation.cc:407] Connection header received
0807 11:35:35.511459 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.511463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.511552 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.511638 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:35.512497 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59664: BlockingRecv error: recv got EOF from 127.0.0.1:59664 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 11:35:35.571800 18474 catalog_manager.cc:1259] Loaded cluster ID: 9f351265cd394930a70df547ee3bc215
I20260807 11:35:35.571965 18474 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6: loading cluster ID for follower catalog manager: success
I20260807 11:35:35.574856 18474 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6: acquiring CA information for follower catalog manager: success
I20260807 11:35:35.576771 18474 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65c2bd7bdd2a4651801730fb3bc2fff6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:35:35.782611 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.776965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.24.103.125:46525)
0807 11:35:35.777210 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.777217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.777440 (+ 223us) server_negotiation.cc:407] Connection header received
0807 11:35:35.777666 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.777671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.777747 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.777838 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.778215 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.778939 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.780413 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.780860 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.780967 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.781605 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.781728 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.781811 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.782096 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.782324 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260807 11:35:35.795315 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.789292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.24.103.124:44689)
0807 11:35:35.789539 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.789546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.789734 (+ 188us) server_negotiation.cc:407] Connection header received
0807 11:35:35.789975 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.789980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.790143 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.790269 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.790733 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.791526 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.793057 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.793460 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.793613 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.794133 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.794658 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.794822 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.794905 (+ 83us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.794996 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 11:35:35.810115 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.802529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.24.103.125:46525)
0807 11:35:35.802824 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.802834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.803653 (+ 819us) server_negotiation.cc:407] Connection header received
0807 11:35:35.804081 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.804085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.804158 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.804234 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.805023 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.805805 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.808055 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.808635 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.808784 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.809222 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.809358 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.809458 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.809694 (+ 236us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.809779 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 11:35:35.813513 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.802215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.24.103.126:37779)
0807 11:35:35.806217 (+ 4002us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.806223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.806243 (+ 20us) server_negotiation.cc:407] Connection header received
0807 11:35:35.806360 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.806366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.806585 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.806792 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.807781 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.808893 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.810850 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.811504 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.811629 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.812245 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.812629 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.812773 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.812977 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.813188 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3856,"thread_start_us":583,"threads_started":1}
I20260807 11:35:35.814296 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:35.803833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.24.103.124:44689)
0807 11:35:35.803990 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:35.803994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:35.807745 (+ 3751us) server_negotiation.cc:407] Connection header received
0807 11:35:35.808017 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:35.808022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:35.808114 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:35.808207 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:35.808799 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.809920 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.812089 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:35.812536 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:35.812631 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:35.813453 (+ 822us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:35.813594 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:35.813725 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:35.813931 (+ 206us) server_negotiation.cc:299] Negotiation successful
0807 11:35:35.814023 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 11:35:35.820684 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 18166
I20260807 11:35:35.822348 18230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:35:35.940997 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 18166
I20260807 11:35:35.967489 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 18242
I20260807 11:35:35.968811 18316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:35:36.078763 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 18242
I20260807 11:35:36.104004 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 18330
I20260807 11:35:36.105224 18405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:35:36.217702 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 18330
2026-08-07T11:35:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5149 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 11:35:36 dist-test-slave-wl2h krb5kdc[18628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 11:35:36 dist-test-slave-wl2h krb5kdc[18628](info): set up 2 sockets
Aug 07 11:35:36 dist-test-slave-wl2h krb5kdc[18628](info): commencing operation
krb5kdc: starting...
W20260807 11:35:37.476408 24989 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/test-user.keytab.
Aug 07 11:35:37 dist-test-slave-wl2h krb5kdc[18628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T11:35:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T11:35:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.103.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
Entry for principal kudu/127.24.103.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab.
I20260807 11:35:37.847666 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38461
--webserver_interface=127.24.103.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38461,127.24.103.125:46365,127.24.103.124:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 11:35:38.203666 18642 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:35:38.204068 18642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:38.204183 18642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:38.204310 18642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:38.211813 18642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:38.211916 18642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:38.212052 18642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:38.212152 18642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:38.212249 18642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:38.222406 18642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38461,127.24.103.125:46365,127.24.103.124:37799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.126.keytab
--principal=kudu/127.24.103.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.126:38461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.103.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:38.223909 18642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:38.225337 18642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:38.238168 18648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:38.238296 18649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:38.238667 18651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:38.239023 18642 server_base.cc:1034] running on GCE node
I20260807 11:35:38.239841 18642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:38.241338 18642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:38.242642 18642 hybrid_clock.cc:648] HybridClock initialized: now 1786102538242573 us; error 34 us; skew 500 ppm
I20260807 11:35:38.243067 18642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:38 dist-test-slave-wl2h krb5kdc[18628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102538, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:38.252585 18642 init.cc:377] Logged in from keytab as kudu/127.24.103.126@KRBTEST.COM (short username kudu)
I20260807 11:35:38.254514 18642 webserver.cc:460] Webserver started at http://127.24.103.126:33033/ using document root <none> and password file <none>
I20260807 11:35:38.255381 18642 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:38.255522 18642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:38.255868 18642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:38.258948 18642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/data/instance:
uuid: "dc53d2b57f8c4b899ed6092d69f4153f"
format_stamp: "Formatted at 2026-08-07 11:35:38 on dist-test-slave-wl2h"
server_key: "73dcb90cd6fb7aca634e5358779e5064"
server_key_iv: "f7dc511de076062e039ba76b9e0ca152"
server_key_version: "encryptionkey@0"
I20260807 11:35:38.259853 18642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/wal/instance:
uuid: "dc53d2b57f8c4b899ed6092d69f4153f"
format_stamp: "Formatted at 2026-08-07 11:35:38 on dist-test-slave-wl2h"
server_key: "73dcb90cd6fb7aca634e5358779e5064"
server_key_iv: "f7dc511de076062e039ba76b9e0ca152"
server_key_version: "encryptionkey@0"
I20260807 11:35:38.265476 18642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 11:35:38.269132 18659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:38.270570 18642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:35:38.270788 18642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/wal
uuid: "dc53d2b57f8c4b899ed6092d69f4153f"
format_stamp: "Formatted at 2026-08-07 11:35:38 on dist-test-slave-wl2h"
server_key: "73dcb90cd6fb7aca634e5358779e5064"
server_key_iv: "f7dc511de076062e039ba76b9e0ca152"
server_key_version: "encryptionkey@0"
I20260807 11:35:38.271054 18642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:38.294066 18642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:38.298530 18642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:38.298833 18642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:38.318583 18642 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.126:38461
I20260807 11:35:38.318601 18721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.126:38461 every 8 connection(s)
I20260807 11:35:38.320041 18642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/data/info.pb
I20260807 11:35:38.324064 18722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:38.324311 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 18642
I20260807 11:35:38.324518 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-0/wal/instance
I20260807 11:35:38.325100 24989 external_mini_cluster.cc:1428] Setting key 59f69326fcd150e0496479725db47a4e
I20260807 11:35:38.334528 18722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:35:38 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102538, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.125@KRBTEST.COM; defaulting to no policy
I20260807 11:35:38.360839 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.339925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.24.103.126:38461)
0807 11:35:38.340698 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:38.340706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:38.340766 (+ 60us) server_negotiation.cc:407] Connection header received
0807 11:35:38.342804 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:38.342810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:38.343119 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:38.343331 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:38.345043 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.346195 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.348528 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.349087 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.349261 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:38.353385 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:38.353393 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:38.353405 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:38.353432 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:38.356244 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:38.356958 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:38.356964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:38.356967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:38.357108 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:38.357537 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:38.357541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:38.357545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:38.358032 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:38.358256 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:38.358815 (+ 559us) server_negotiation.cc:299] Negotiation successful
0807 11:35:38.359278 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":531,"thread_start_us":368,"threads_started":1}
I20260807 11:35:38.360860 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.338472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38461 (local address 127.0.0.1:56090)
0807 11:35:38.339468 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.339516 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:38.342396 (+ 2880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:38.343407 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:38.343417 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:38.344248 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:38.344795 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:38.344828 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.346399 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.346407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:38.348378 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:38.348396 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.349352 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.349363 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:38.349620 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:38.350336 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:38.350369 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:38.353153 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:38.356388 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:38.356398 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:38.356411 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:38.356813 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:38.357248 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:38.357254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:38.357258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:38.357414 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:38.358188 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:38.358197 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:38.358407 (+ 210us) client_negotiation.cc:769] Sending connection context
0807 11:35:38.358660 (+ 253us) client_negotiation.cc:240] Negotiation successful
0807 11:35:38.359402 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":213,"thread_start_us":504,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.125@KRBTEST.COM" created.
I20260807 11:35:38.364454 18722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:35:38.365890 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.365304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36248)
0807 11:35:38.365542 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.365700 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260807 11:35:38.366264 18661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
I20260807 11:35:38.368232 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.367701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36256)
0807 11:35:38.367903 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.368011 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 11:35:38.368467 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:46365: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
Entry for principal kudu/127.24.103.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab.
I20260807 11:35:38.407405 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:46365
--webserver_interface=127.24.103.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38461,127.24.103.125:46365,127.24.103.124:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:35:38.417871 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } attempt: 1
I20260807 11:35:38.419287 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.418658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36258)
0807 11:35:38.418926 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.419103 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260807 11:35:38.420528 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.420078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36266)
0807 11:35:38.420255 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.420351 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 11:35:38.420758 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:46365: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
I20260807 11:35:38.489076 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } attempt: 2
I20260807 11:35:38.490736 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.490184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36282)
0807 11:35:38.490476 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.490572 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260807 11:35:38.491904 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.491499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36290)
0807 11:35:38.491670 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.491760 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 11:35:38.492110 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:46365: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
I20260807 11:35:38.583377 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } attempt: 3
I20260807 11:35:38.584625 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.584184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36298)
0807 11:35:38.584398 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.584489 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 11:35:38.585702 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.585335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36308)
0807 11:35:38.585485 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.585562 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 11:35:38.585888 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:46365: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
I20260807 11:35:38.729158 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } attempt: 4
I20260807 11:35:38.730613 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.729964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36320)
0807 11:35:38.730315 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.730404 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260807 11:35:38.731765 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.731387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36324)
0807 11:35:38.731545 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.731637 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 11:35:38.731956 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.125:46365: Network error: Client connection negotiation failed: client connection to 127.24.103.125:46365: connect: Connection refused (error 111)
W20260807 11:35:38.763149 18730 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:35:38.763547 18730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:38.763679 18730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:38.763823 18730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:38.771376 18730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:38.771512 18730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:38.771646 18730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:38.771740 18730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:38.771853 18730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:38.782254 18730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38461,127.24.103.125:46365,127.24.103.124:37799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.125.keytab
--principal=kudu/127.24.103.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.125:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.103.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:38.783707 18730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:38.785096 18730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:38.797324 18736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:38.797524 18737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:38.797489 18739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:38.798378 18730 server_base.cc:1034] running on GCE node
I20260807 11:35:38.799283 18730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:38.800760 18730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:38.802086 18730 hybrid_clock.cc:648] HybridClock initialized: now 1786102538801986 us; error 56 us; skew 500 ppm
I20260807 11:35:38.802492 18730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:38 dist-test-slave-wl2h krb5kdc[18628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102538, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:38.805744 18730 init.cc:377] Logged in from keytab as kudu/127.24.103.125@KRBTEST.COM (short username kudu)
I20260807 11:35:38.807271 18730 webserver.cc:460] Webserver started at http://127.24.103.125:40175/ using document root <none> and password file <none>
I20260807 11:35:38.807989 18730 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:38.808130 18730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:38.808506 18730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:38.811576 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/data/instance:
uuid: "08fa8c7346224fccb899dd087eafbafd"
format_stamp: "Formatted at 2026-08-07 11:35:38 on dist-test-slave-wl2h"
server_key: "90ce2e2bbab57f0c8aa325da8563d948"
server_key_iv: "1cfcd2122dd25f188eb786f8e705569f"
server_key_version: "encryptionkey@0"
I20260807 11:35:38.812500 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/wal/instance:
uuid: "08fa8c7346224fccb899dd087eafbafd"
format_stamp: "Formatted at 2026-08-07 11:35:38 on dist-test-slave-wl2h"
server_key: "90ce2e2bbab57f0c8aa325da8563d948"
server_key_iv: "1cfcd2122dd25f188eb786f8e705569f"
server_key_version: "encryptionkey@0"
I20260807 11:35:38.817651 18730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 11:35:38.821156 18747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:38.822757 18730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 11:35:38.822965 18730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/wal
uuid: "08fa8c7346224fccb899dd087eafbafd"
format_stamp: "Formatted at 2026-08-07 11:35:38 on dist-test-slave-wl2h"
server_key: "90ce2e2bbab57f0c8aa325da8563d948"
server_key_iv: "1cfcd2122dd25f188eb786f8e705569f"
server_key_version: "encryptionkey@0"
I20260807 11:35:38.823300 18730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:38.865396 18730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:38.870289 18730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:38.870613 18730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:38.891337 18730 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.125:46365
I20260807 11:35:38.891391 18802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.125:46365 every 8 connection(s)
I20260807 11:35:38.892855 18730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/data/info.pb
I20260807 11:35:38.894273 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 18730
I20260807 11:35:38.894505 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-1/wal/instance
I20260807 11:35:38.895169 24989 external_mini_cluster.cc:1428] Setting key bae40401909f5526a0890ff0af49f362
I20260807 11:35:38.897825 18803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:38.907893 18803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:35:38 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102538, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.103.124@KRBTEST.COM; defaulting to no policy
I20260807 11:35:38.933667 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.911714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56106 (local address 127.24.103.126:38461)
0807 11:35:38.912573 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:38.912580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:38.913036 (+ 456us) server_negotiation.cc:407] Connection header received
0807 11:35:38.916202 (+ 3166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:38.916207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:38.916282 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:38.916384 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:38.918596 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.919500 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.922014 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.922432 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.922537 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:38.927071 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:38.927084 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:38.927087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:38.927135 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:38.930384 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:38.931245 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:38.931251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:38.931256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:38.931381 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:38.931834 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:38.931846 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:38.931848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:38.932148 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:38.932338 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:38.933037 (+ 699us) server_negotiation.cc:299] Negotiation successful
0807 11:35:38.933277 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":562,"threads_started":1}
I20260807 11:35:38.934747 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.912064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38461 (local address 127.0.0.1:56106)
0807 11:35:38.912870 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.912918 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:38.915810 (+ 2892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:38.916664 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:38.916677 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:38.917476 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:38.918341 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:38.918368 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.919683 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.919691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:38.921822 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:38.921846 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.922728 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.922738 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:38.923095 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:38.923868 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:38.923903 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:38.926755 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:38.930543 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:38.930552 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:38.930569 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:38.931115 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:38.931507 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:38.931512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:38.931516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:38.931705 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:38.932287 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:38.932295 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:38.932661 (+ 366us) client_negotiation.cc:769] Sending connection context
0807 11:35:38.932931 (+ 270us) client_negotiation.cc:240] Negotiation successful
0807 11:35:38.933468 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":346,"threads_started":1}
I20260807 11:35:38.936602 18803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.103.124@KRBTEST.COM" created.
Aug 07 11:35:38 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102538, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260807 11:35:38.952198 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.937352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36336)
0807 11:35:38.937596 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.937613 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:38.937699 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:38.938790 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:38.938794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:38.939271 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:38.939401 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:38.939409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.940761 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.940766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:38.941588 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:38.941597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.942395 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.942400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:38.942577 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:38.943340 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:38.943357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:38.945703 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:38.949327 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:38.949333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:38.949337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:38.949753 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:38.950173 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:38.950180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:38.950184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:38.950333 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:38.950979 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:38.950989 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:38.951196 (+ 207us) client_negotiation.cc:769] Sending connection context
0807 11:35:38.951348 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 11:35:38.951705 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260807 11:35:38.952339 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.937706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.24.103.125:46365)
0807 11:35:38.938390 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:38.938396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:38.938420 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:35:38.938460 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:38.938465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:38.938642 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:38.938798 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:38.939577 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.940624 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.941723 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:38.942255 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:38.942389 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:38.945929 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:38.945941 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:38.945953 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:38.945981 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:38.949153 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:38.949901 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:38.949918 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:38.949920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:38.950041 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:38.950448 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:38.950456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:38.950460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:38.950844 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:38.951144 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:38.951505 (+ 361us) server_negotiation.cc:299] Negotiation successful
0807 11:35:38.951846 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":537,"thread_start_us":384,"threads_started":1}
I20260807 11:35:38.954981 18803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:35:38.956378 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.955890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33204)
0807 11:35:38.956077 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.956225 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260807 11:35:38.956737 18749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
I20260807 11:35:38.958426 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:38.958070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33216)
0807 11:35:38.958231 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:38.958310 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 11:35:38.958752 18803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:37799: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
Entry for principal kudu/127.24.103.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab.
I20260807 11:35:38.983085 24989 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska8aRx7/build/asan/bin/kudu
/tmp/dist-test-taska8aRx7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:37799
--webserver_interface=127.24.103.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.103.84:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.103.126:38461,127.24.103.125:46365,127.24.103.124:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 11:35:39.008131 18803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } attempt: 1
I20260807 11:35:39.009469 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.008946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33218)
0807 11:35:39.009168 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.009306 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 11:35:39.010525 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.010161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33232)
0807 11:35:39.010344 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.010419 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 11:35:39.010730 18803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:37799: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
I20260807 11:35:39.031172 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } attempt: 5
Aug 07 11:35:39 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102538, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260807 11:35:39.046972 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.031990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36346)
0807 11:35:39.032205 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.032229 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:39.032352 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:39.032780 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:39.032784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:39.033347 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:39.033533 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.033542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.034957 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.034961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.035948 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.035960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.036965 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.036969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.037140 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.037818 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:39.037836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:39.040279 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:39.044458 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.044462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.044465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.044876 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.045322 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.045327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.045330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.045508 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.046200 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:39.046204 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:39.046343 (+ 139us) client_negotiation.cc:769] Sending connection context
0807 11:35:39.046471 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 11:35:39.046715 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 11:35:39.047209 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.032107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.24.103.125:46365)
0807 11:35:39.032316 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.032322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.032347 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:35:39.032477 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.032484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.032584 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.032713 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.033714 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.034795 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.036096 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.036789 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.036928 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.040495 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.040503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.040507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.040560 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.044284 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.045005 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.045011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.045015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.045171 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.045664 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.045668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.045672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.046051 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.046209 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.046627 (+ 418us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.046863 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 11:35:39.048420 18722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:35:39.049880 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.049310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33236)
0807 11:35:39.049555 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.049673 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 11:35:39.051231 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.050692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33242)
0807 11:35:39.050880 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.050997 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 11:35:39.051548 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:37799: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
I20260807 11:35:39.079035 18803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } attempt: 2
I20260807 11:35:39.080266 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.079792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33244)
0807 11:35:39.080005 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.080100 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260807 11:35:39.081256 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.080928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33252)
0807 11:35:39.081072 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.081143 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 11:35:39.081513 18803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:37799: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
I20260807 11:35:39.102869 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } attempt: 1
I20260807 11:35:39.104234 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.103644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33254)
0807 11:35:39.103904 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.104019 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260807 11:35:39.105350 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.104959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33262)
0807 11:35:39.105124 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.105219 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 11:35:39.105736 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:37799: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
I20260807 11:35:39.172784 18803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } attempt: 3
I20260807 11:35:39.173980 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } attempt: 2
I20260807 11:35:39.174127 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.173542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33272)
0807 11:35:39.173797 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.173914 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 11:35:39.175264 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.174632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33278)
0807 11:35:39.174905 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.175044 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260807 11:35:39.175532 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.174925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33280)
0807 11:35:39.175262 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.175389 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
W20260807 11:35:39.175855 18803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:37799: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
I20260807 11:35:39.176381 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.176011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33284)
0807 11:35:39.176173 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.176258 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 11:35:39.176645 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:37799: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
I20260807 11:35:39.282935 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } attempt: 3
I20260807 11:35:39.284312 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.283803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33298)
0807 11:35:39.284026 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.284133 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 11:35:39.285451 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.285012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33304)
0807 11:35:39.285267 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.285345 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260807 11:35:39.285670 18722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:37799: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
I20260807 11:35:39.319164 18803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } attempt: 4
I20260807 11:35:39.320482 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.319855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33308)
0807 11:35:39.320127 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.320235 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 11:35:39.321671 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.321285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33320)
0807 11:35:39.321445 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.321532 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 11:35:39.321979 18803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.103.124:37799: Network error: Client connection negotiation failed: client connection to 127.24.103.124:37799: connect: Connection refused (error 111)
W20260807 11:35:39.341547 18813 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 11:35:39.341869 18813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 11:35:39.342017 18813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 11:35:39.342154 18813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 11:35:39.349560 18813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 11:35:39.349654 18813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 11:35:39.349747 18813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 11:35:39.349843 18813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 11:35:39.349952 18813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 11:35:39.360178 18813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.103.84:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.103.126:38461,127.24.103.125:46365,127.24.103.124:37799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/krb5kdc/kudu_127.24.103.124.keytab
--principal=kudu/127.24.103.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.103.124:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.103.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 10:18:12 UTC on ccycloud.quasar-kutpcu.root.comops.site
build id 4572147
ASAN enabled
I20260807 11:35:39.361508 18813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 11:35:39.362880 18813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 11:35:39.374764 18818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:39.374787 18819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 11:35:39.375617 18821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 11:35:39.375725 18813 server_base.cc:1034] running on GCE node
I20260807 11:35:39.376972 18813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 11:35:39.379134 18813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 11:35:39.380385 18813 hybrid_clock.cc:648] HybridClock initialized: now 1786102539380318 us; error 44 us; skew 500 ppm
I20260807 11:35:39.380796 18813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 11:35:39 dist-test-slave-wl2h krb5kdc[18628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102539, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 11:35:39.384584 18813 init.cc:377] Logged in from keytab as kudu/127.24.103.124@KRBTEST.COM (short username kudu)
I20260807 11:35:39.386062 18813 webserver.cc:460] Webserver started at http://127.24.103.124:34043/ using document root <none> and password file <none>
I20260807 11:35:39.386780 18813 fs_manager.cc:359] Metadata directory not provided
I20260807 11:35:39.386915 18813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 11:35:39.387293 18813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 11:35:39.390256 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/data/instance:
uuid: "be4ef70dbd6b4893bf7e43d279f3e285"
format_stamp: "Formatted at 2026-08-07 11:35:39 on dist-test-slave-wl2h"
server_key: "e72a94be6914f103e00523e36a53d111"
server_key_iv: "aa1a28bca3e0d226c3fd8b6aa07f37a7"
server_key_version: "encryptionkey@0"
I20260807 11:35:39.391300 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/wal/instance:
uuid: "be4ef70dbd6b4893bf7e43d279f3e285"
format_stamp: "Formatted at 2026-08-07 11:35:39 on dist-test-slave-wl2h"
server_key: "e72a94be6914f103e00523e36a53d111"
server_key_iv: "aa1a28bca3e0d226c3fd8b6aa07f37a7"
server_key_version: "encryptionkey@0"
I20260807 11:35:39.397223 18813 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 11:35:39.400815 18828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 11:35:39.402279 18813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 11:35:39.402478 18813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/data,/tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/wal
uuid: "be4ef70dbd6b4893bf7e43d279f3e285"
format_stamp: "Formatted at 2026-08-07 11:35:39 on dist-test-slave-wl2h"
server_key: "e72a94be6914f103e00523e36a53d111"
server_key_iv: "aa1a28bca3e0d226c3fd8b6aa07f37a7"
server_key_version: "encryptionkey@0"
I20260807 11:35:39.402719 18813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 11:35:39.422261 18813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 11:35:39.426447 18813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 11:35:39.426738 18813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 11:35:39.446846 18813 rpc_server.cc:307] RPC server started. Bound to: 127.24.103.124:37799
I20260807 11:35:39.446853 18884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.103.124:37799 every 8 connection(s)
I20260807 11:35:39.448297 18813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/data/info.pb
I20260807 11:35:39.449942 24989 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska8aRx7/build/asan/bin/kudu as pid 18813
I20260807 11:35:39.450150 24989 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska8aRx7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786102153172159-24989-0/minicluster-data/master-2/wal/instance
I20260807 11:35:39.450757 24989 external_mini_cluster.cc:1428] Setting key cd00be94433edb29ca2f09c94079fb3b
I20260807 11:35:39.453069 18885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 11:35:39.461489 18885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:35:39.462867 18722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } attempt: 4
Aug 07 11:35:39 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
Aug 07 11:35:39 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102538, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.126@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260807 11:35:39.481556 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.452611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.24.103.124:37799)
0807 11:35:39.453401 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.453412 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.453452 (+ 40us) server_negotiation.cc:407] Connection header received
0807 11:35:39.455103 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.455123 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.455496 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.455822 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.457706 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.459067 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.460866 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.461536 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.461686 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.467735 (+ 6049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.467749 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.467773 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.467841 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.473312 (+ 5471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.474542 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.474556 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.474570 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.474929 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.475555 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.475563 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.475569 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.476572 (+ 1003us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.476959 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.478933 (+ 1974us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.479607 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":369,"threads_started":1}
I20260807 11:35:39.485937 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.463549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33334)
0807 11:35:39.463872 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.463898 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:39.464030 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:39.464976 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:39.464985 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:39.465748 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:39.466027 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.466042 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.468561 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.468571 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.470485 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.470506 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.471978 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.471987 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.472269 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.472967 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:39.472999 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:39.477104 (+ 4105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:39.482745 (+ 5641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.482750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.482753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.483354 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.483891 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.483896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.483900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.484076 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.484793 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:39.484799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:39.484984 (+ 185us) client_negotiation.cc:769] Sending connection context
0807 11:35:39.485189 (+ 205us) client_negotiation.cc:240] Negotiation successful
0807 11:35:39.485509 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":1}
I20260807 11:35:39.485963 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.463672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.24.103.124:37799)
0807 11:35:39.464592 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.464597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.464622 (+ 25us) server_negotiation.cc:407] Connection header received
0807 11:35:39.464667 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.464681 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.464757 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.466215 (+ 1458us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.466546 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.468317 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.470755 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.471754 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.472666 (+ 912us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.477492 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.477501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.477509 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.477552 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.482566 (+ 5014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.483543 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.483548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.483552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.483732 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.484205 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.484210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.484214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.484563 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.484820 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.485287 (+ 467us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.485506 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":759,"thread_start_us":1091,"threads_started":1}
Aug 07 11:35:39 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102539, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:35:39.492902 18722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f: Bootstrap starting.
I20260807 11:35:39.497536 18722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f: Neither blocks nor log segments found. Creating new log.
Aug 07 11:35:39 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.126@KRBTEST.COM
I20260807 11:35:39.499142 18722 log.cc:824] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:39.499402 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.464649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.126:38461 (local address 127.0.0.1:56108)
0807 11:35:39.482621 (+ 17972us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.482666 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:39.482865 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:39.483297 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:39.483304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:39.484187 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:39.484593 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.484606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.486350 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.486356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.487907 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.487925 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.488800 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.488807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.488997 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.489712 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:39.489740 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:39.492476 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:39.496497 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.496512 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.496526 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.496912 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.497306 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.497312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.497317 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.497458 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.498148 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:39.498156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:39.498357 (+ 201us) client_negotiation.cc:769] Sending connection context
0807 11:35:39.498544 (+ 187us) client_negotiation.cc:240] Negotiation successful
0807 11:35:39.498903 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17758,"mutex_wait_us":6,"thread_start_us":17583,"threads_started":1}
I20260807 11:35:39.499426 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.464680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.24.103.126:38461)
0807 11:35:39.482381 (+ 17701us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.482389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.482772 (+ 383us) server_negotiation.cc:407] Connection header received
0807 11:35:39.482994 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.482999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.483117 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.483236 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.484926 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.486046 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.488113 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.488597 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.488742 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.492693 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.492699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.492702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.492733 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.496342 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.497047 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.497052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.497056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.497184 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.497581 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.497588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.497599 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.497999 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.498177 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.498662 (+ 485us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.498902 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17542,"thread_start_us":17385,"threads_started":1}
I20260807 11:35:39.501034 18885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } has no permanent_uuid. Determining permanent_uuid...
I20260807 11:35:39.504552 18722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f: No bootstrap required, opened a new log
I20260807 11:35:39.508970 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.486394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.24.103.126:38461)
0807 11:35:39.487629 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.487636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.487662 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:35:39.487712 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.487719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.487797 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.487909 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.490623 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.491891 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.493649 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.494439 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.494596 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.500804 (+ 6208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.500812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.500816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.501528 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.505172 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.505919 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.505926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.505930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.506072 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.506499 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.506506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.506515 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.506951 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.508338 (+ 1387us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.508475 (+ 137us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.508667 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1094,"thread_start_us":1338,"threads_started":1}
Aug 07 11:35:39 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102539, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
I20260807 11:35:39.514500 18722 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } }
I20260807 11:35:39.514870 18722 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc53d2b57f8c4b899ed6092d69f4153f, State: Initialized, Role: FOLLOWER
I20260807 11:35:39.515478 18722 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } }
I20260807 11:35:39.516812 18895 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } } }
I20260807 11:35:39.517326 18895 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:39.518435 18722 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:39.519302 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.501741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.125:46365 (local address 127.0.0.1:36360)
0807 11:35:39.502513 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.502533 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:39.502652 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:39.503085 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:39.503089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:39.503584 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:39.503781 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.503793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.505247 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.505254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.506592 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.506608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.507642 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.507650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.507865 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.508540 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:39.508564 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:39.511246 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:39.516184 (+ 4938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.516190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.516194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.516800 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.517319 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.517323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.517327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.517511 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.518244 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:39.518252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:39.518415 (+ 163us) client_negotiation.cc:769] Sending connection context
0807 11:35:39.518559 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 11:35:39.518868 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643}
I20260807 11:35:39.520023 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.502120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.24.103.125:46365)
0807 11:35:39.502308 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.502312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.502640 (+ 328us) server_negotiation.cc:407] Connection header received
0807 11:35:39.502780 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.502787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.502888 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.503006 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.503982 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.505006 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.506795 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.507473 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.507599 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.511523 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.511533 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.511540 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.511596 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.516003 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.516964 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.516971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.516975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.517161 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.517645 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.517656 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.517660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.518092 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.518257 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.519223 (+ 966us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.519470 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":76}
I20260807 11:35:39.521215 18885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 11:35:39 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.103.125@KRBTEST.COM
Aug 07 11:35:39 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102539, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.124@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260807 11:35:39.535596 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.511077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36374 (local address 127.24.103.125:46365)
0807 11:35:39.515644 (+ 4567us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.515651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.515677 (+ 26us) server_negotiation.cc:407] Connection header received
0807 11:35:39.515732 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.515737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.515826 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.515957 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.517705 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.518764 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.520541 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.521137 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.521280 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.528039 (+ 6759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.528049 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.528056 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.528125 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.531892 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.532842 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.532848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.532851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.533026 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.533559 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.533564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.533568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.533931 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.534237 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.534849 (+ 612us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.535129 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4398,"thread_start_us":4163,"threads_started":1}
I20260807 11:35:39.539597 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.522121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33350)
0807 11:35:39.522315 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.522334 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:39.522454 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:39.523378 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:39.523383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:39.523890 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:39.524114 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.524129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.525579 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.525583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.526960 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.526977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.528145 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.528152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.528350 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.529127 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:39.529144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:39.532108 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:39.536592 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.536597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.536602 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.537157 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.537651 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.537655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.537658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.537813 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.538552 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:39.538558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:39.538711 (+ 153us) client_negotiation.cc:769] Sending connection context
0807 11:35:39.538874 (+ 163us) client_negotiation.cc:240] Negotiation successful
0807 11:35:39.539232 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 11:35:39.539628 18909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 11:35:39.539748 18909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 11:35:39.540427 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.522699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.24.103.124:37799)
0807 11:35:39.522905 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.522910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.522929 (+ 19us) server_negotiation.cc:407] Connection header received
0807 11:35:39.522977 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.522983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.523101 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.523204 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.524278 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.525411 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.527159 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.527830 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.527930 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.532279 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.532285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.532289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.532325 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.536233 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.537358 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.537363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.537366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.537524 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.537930 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.537934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.537938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.538348 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.538503 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.539898 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.540118 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 11:35:39.546357 18885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285: Bootstrap starting.
I20260807 11:35:39.547231 24989 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 11:35:39.537531 18642 master_runner.cc:186] Error getting master registration for 127.24.103.125:46365: OK, instance_id { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" instance_seqno: 1786102538885086 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:35:39.548094 18642 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 11:35:39.550431 18885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:39.550400 18688 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dc53d2b57f8c4b899ed6092d69f4153f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56110
I20260807 11:35:39.550812 18688 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 11:35:39.551052 18688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [term 0 FOLLOWER]: Advancing to term 1
W20260807 11:35:39.551739 18666 tablet.cc:2369] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:35:39.551793 18885 log.cc:824] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285: Log is configured to *not* fsync() on all Append() calls
I20260807 11:35:39.553678 18688 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } }
I20260807 11:35:39.555526 18688 leader_election.cc:288] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [CANDIDATE]: Term 1 election: Requested vote from peers 08fa8c7346224fccb899dd087eafbafd (127.24.103.125:46365), be4ef70dbd6b4893bf7e43d279f3e285 (127.24.103.124:37799)
I20260807 11:35:39.561721 18885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285: No bootstrap required, opened a new log
I20260807 11:35:39.570297 18885 raft_consensus.cc:348] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } }
I20260807 11:35:39.571074 18885 raft_consensus.cc:729] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4ef70dbd6b4893bf7e43d279f3e285, State: Initialized, Role: FOLLOWER
I20260807 11:35:39.571853 18885 consensus_queue.cc:260] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } }
I20260807 11:35:39.589100 18918 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } } }
I20260807 11:35:39.589747 18918 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:39.591174 18885 sys_catalog.cc:564] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:39.591236 18860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "be4ef70dbd6b4893bf7e43d279f3e285"
I20260807 11:35:39.591874 18860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:39.595561 18860 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc53d2b57f8c4b899ed6092d69f4153f in term 1.
I20260807 11:35:39.596388 18662 leader_election.cc:302] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be4ef70dbd6b4893bf7e43d279f3e285, dc53d2b57f8c4b899ed6092d69f4153f; no voters:
I20260807 11:35:39.596971 18895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 11:35:39.613301 18895 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [term 1 LEADER]: Becoming Leader. State: Replica: dc53d2b57f8c4b899ed6092d69f4153f, State: Running, Role: LEADER
I20260807 11:35:39.614151 18895 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } }
I20260807 11:35:39.620675 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.569757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.24.103.126:38461)
0807 11:35:39.569998 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.570005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.583829 (+ 13824us) server_negotiation.cc:407] Connection header received
0807 11:35:39.584184 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.584193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.584320 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.584414 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.591478 (+ 7064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.592611 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.597156 (+ 4545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.597974 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.598093 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.610427 (+ 12334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.610437 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.610445 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.610507 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.615153 (+ 4646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.616566 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.616573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.616578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.616744 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.617613 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.617620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.617623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.618060 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.618331 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.619498 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.619990 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 11:35:39.621152 18803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } attempt: 5
I20260807 11:35:39.626878 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.570220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.24.103.124:37799)
0807 11:35:39.570388 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.570394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.588228 (+ 17834us) server_negotiation.cc:407] Connection header received
0807 11:35:39.588572 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.588580 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.588724 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.588839 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.599552 (+ 10713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.600629 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.603929 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.604778 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.604891 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.611375 (+ 6484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.611384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.611392 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.611442 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.615082 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.623329 (+ 8247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.623338 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.623341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.623576 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.624342 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.624347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.624351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.624741 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.625061 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.626183 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.626462 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Aug 07 11:35:39 dist-test-slave-wl2h krb5kdc[18628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786102538, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.103.125@KRBTEST.COM for kudu/127.24.103.124@KRBTEST.COM
I20260807 11:35:39.636619 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.584086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.24.103.125:46365)
0807 11:35:39.584375 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.584385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.608285 (+ 23900us) server_negotiation.cc:407] Connection header received
0807 11:35:39.610543 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.610552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.610660 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.610762 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.612830 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.613974 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.616296 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.617009 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.617123 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.627182 (+ 10059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.627190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.627198 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.627248 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.631379 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.632336 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.632341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.632344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.632601 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.633471 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.633500 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.633506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.634156 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.634555 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.635696 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.636051 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 11:35:39.645396 18921 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc53d2b57f8c4b899ed6092d69f4153f. Latest consensus state: current_term: 1 leader_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } } }
I20260807 11:35:39.645901 18921 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:39.647331 18926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 11:35:39.649067 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.621863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.103.124:37799 (local address 127.0.0.1:33374)
0807 11:35:39.622361 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 11:35:39.622387 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 11:35:39.622520 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 11:35:39.624528 (+ 2008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 11:35:39.624536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 11:35:39.625262 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0807 11:35:39.625599 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.625615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.628359 (+ 2744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.628368 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.630688 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 11:35:39.630713 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.632404 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.632414 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 11:35:39.632887 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.633663 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 11:35:39.633698 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 11:35:39.637837 (+ 4139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 11:35:39.644587 (+ 6750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.644607 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.644612 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.645238 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.645769 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 11:35:39.645776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 11:35:39.645781 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 11:35:39.646156 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 11:35:39.646955 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 11:35:39.646961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 11:35:39.647163 (+ 202us) client_negotiation.cc:769] Sending connection context
0807 11:35:39.648044 (+ 881us) client_negotiation.cc:240] Negotiation successful
0807 11:35:39.648486 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335}
I20260807 11:35:39.650569 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:39.622805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.24.103.124:37799)
0807 11:35:39.623070 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.623079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.623103 (+ 24us) server_negotiation.cc:407] Connection header received
0807 11:35:39.623174 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.623182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.623282 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.623412 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.625934 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.628143 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.630946 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:39.631952 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:39.632133 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:39.639748 (+ 7615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 11:35:39.639759 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 11:35:39.639767 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 11:35:39.639837 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0807 11:35:39.644352 (+ 4515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.645410 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.645419 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.645425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.645611 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 11:35:39.646317 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 11:35:39.646323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 11:35:39.646327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 11:35:39.646734 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 11:35:39.646872 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:39.649543 (+ 2671us) server_negotiation.cc:299] Negotiation successful
0807 11:35:39.650047 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 11:35:39.651942 18926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 11:35:39.662351 18803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd: Bootstrap starting.
I20260807 11:35:39.671361 18803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd: Neither blocks nor log segments found. Creating new log.
I20260807 11:35:39.673362 18803 log.cc:824] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd: Log is configured to *not* fsync() on all Append() calls
W20260807 11:35:39.679073 18834 tablet.cc:2369] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 11:35:39.684958 18803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd: No bootstrap required, opened a new log
I20260807 11:35:39.686280 18860 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [term 1 FOLLOWER]: Refusing update from remote peer dc53d2b57f8c4b899ed6092d69f4153f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:35:39.687409 18895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [LEADER]: Connected to new peer: Peer: permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:35:39.708854 18918 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc53d2b57f8c4b899ed6092d69f4153f. Latest consensus state: current_term: 1 leader_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } } }
I20260807 11:35:39.709329 18918 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:39.714134 18928 mvcc.cc:205] Tried to move back new op lower bound from 7315876002502127616 to 7315876002266845184. Current Snapshot: MvccSnapshot[applied={T|T < 7315876002502127616}]
I20260807 11:35:39.714593 18932 mvcc.cc:205] Tried to move back new op lower bound from 7315876002502127616 to 7315876002266845184. Current Snapshot: MvccSnapshot[applied={T|T < 7315876002502127616}]
I20260807 11:35:39.723002 18921 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } } }
I20260807 11:35:39.723407 18921 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:39.728453 18926 catalog_manager.cc:1347] Generated new cluster ID: 656a28092b29432b8f032704e443d250
I20260807 11:35:39.728569 18926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 11:35:39.729087 18918 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } } }
I20260807 11:35:39.729465 18918 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:39.740073 18803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } }
I20260807 11:35:39.740754 18803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fa8c7346224fccb899dd087eafbafd, State: Initialized, Role: FOLLOWER
I20260807 11:35:39.741516 18803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } }
I20260807 11:35:39.749012 18938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } } }
I20260807 11:35:39.749603 18938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:39.752311 18803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [sys.catalog]: configured and running, proceeding with master startup.
I20260807 11:35:39.752815 18775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [term 0 FOLLOWER]: Advancing to term 1
I20260807 11:35:39.756418 18776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "08fa8c7346224fccb899dd087eafbafd"
I20260807 11:35:39.758802 18775 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [term 1 FOLLOWER]: Refusing update from remote peer dc53d2b57f8c4b899ed6092d69f4153f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 11:35:39.763029 18776 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc53d2b57f8c4b899ed6092d69f4153f in term 1.
I20260807 11:35:39.770035 18895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 11:35:39.772172 18944 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 11:35:39.778898 18813 master_runner.cc:186] Error getting master registration for 127.24.103.125:46365: OK, instance_id { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" instance_seqno: 1786102538885086 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 11:35:39.804905 18813 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 11:35:39.832594 18938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc53d2b57f8c4b899ed6092d69f4153f. Latest consensus state: current_term: 1 leader_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } } }
I20260807 11:35:39.833065 18938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:39.834542 18895 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } } }
I20260807 11:35:39.835047 18895 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f [sys.catalog]: This master's current role is: LEADER
I20260807 11:35:39.836732 18954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc53d2b57f8c4b899ed6092d69f4153f" member_type: VOTER last_known_addr { host: "127.24.103.126" port: 38461 } } peers { permanent_uuid: "08fa8c7346224fccb899dd087eafbafd" member_type: VOTER last_known_addr { host: "127.24.103.125" port: 46365 } } peers { permanent_uuid: "be4ef70dbd6b4893bf7e43d279f3e285" member_type: VOTER last_known_addr { host: "127.24.103.124" port: 37799 } } }
I20260807 11:35:39.837253 18954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd [sys.catalog]: This master's current role is: FOLLOWER
I20260807 11:35:39.842890 18926 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 11:35:39.844946 18926 catalog_manager.cc:1504] Loading token signing keys...
I20260807 11:35:39.866488 18958 catalog_manager.cc:1259] Loaded cluster ID: 656a28092b29432b8f032704e443d250
I20260807 11:35:39.866633 18958 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd: loading cluster ID for follower catalog manager: success
I20260807 11:35:39.870469 18958 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd: acquiring CA information for follower catalog manager: success
W20260807 11:35:39.871838 18958 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260807 11:35:39.872578 18926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc53d2b57f8c4b899ed6092d69f4153f: Generated new TSK 0
I20260807 11:35:39.873569 18926 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 11:35:39.985299 18892 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:39.969764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56130 (local address 127.24.103.126:38461)
0807 11:35:39.970220 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.970226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.981321 (+ 11095us) server_negotiation.cc:407] Connection header received
0807 11:35:39.983176 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.983184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.983316 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.983416 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.984965 (+ 1549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56130: BlockingRecv error: recv got EOF from 127.0.0.1:56130 (error 108)
Metrics: {"server-negotiator.queue_time_us":309}
W20260807 11:35:39.989343 18888 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:39.969506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.24.103.124:37799)
0807 11:35:39.970029 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.970039 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.987420 (+ 17381us) server_negotiation.cc:407] Connection header received
0807 11:35:39.987688 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.987694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.987784 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.987857 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.989054 (+ 1197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33382: BlockingRecv error: recv got EOF from 127.0.0.1:33382 (error 108)
Metrics: {"server-negotiator.queue_time_us":345}
W20260807 11:35:39.990087 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:39.982026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.24.103.125:46365)
0807 11:35:39.982244 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:39.982251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:39.987565 (+ 5314us) server_negotiation.cc:407] Connection header received
0807 11:35:39.988398 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:39.988405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:39.988544 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:39.988631 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:39.989777 (+ 1146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36390: BlockingRecv error: recv got EOF from 127.0.0.1:36390 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260807 11:35:40.003810 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:40.001226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.24.103.125:46365)
0807 11:35:40.001433 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:40.001440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:40.001995 (+ 555us) server_negotiation.cc:407] Connection header received
0807 11:35:40.002260 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:40.002267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:40.002429 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:40.002569 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:40.003528 (+ 959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36406: BlockingRecv error: recv got EOF from 127.0.0.1:36406 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260807 11:35:40.020586 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:40.017809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.24.103.125:46365)
0807 11:35:40.018074 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:40.018081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:40.018691 (+ 610us) server_negotiation.cc:407] Connection header received
0807 11:35:40.018919 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:40.018928 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:40.019104 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:40.019242 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:40.020233 (+ 991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36410: BlockingRecv error: recv got EOF from 127.0.0.1:36410 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260807 11:35:40.029698 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:40.027046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.24.103.125:46365)
0807 11:35:40.027258 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:40.027264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:40.027926 (+ 662us) server_negotiation.cc:407] Connection header received
0807 11:35:40.028138 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:40.028145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:40.028275 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:40.028408 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:40.029398 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36412: BlockingRecv error: recv got EOF from 127.0.0.1:36412 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 11:35:40.041594 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:40.038905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36414 (local address 127.24.103.125:46365)
0807 11:35:40.039250 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:40.039258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:40.039825 (+ 567us) server_negotiation.cc:407] Connection header received
0807 11:35:40.040071 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:40.040080 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:40.040199 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:40.040330 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:40.041300 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36414: BlockingRecv error: recv got EOF from 127.0.0.1:36414 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
W20260807 11:35:40.057683 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:40.055283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.24.103.125:46365)
0807 11:35:40.055476 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:40.055485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:40.056028 (+ 543us) server_negotiation.cc:407] Connection header received
0807 11:35:40.056257 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:40.056264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:40.056386 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:40.056544 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:40.057437 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36422: BlockingRecv error: recv got EOF from 127.0.0.1:36422 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 11:35:40.087432 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:40.084909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.24.103.125:46365)
0807 11:35:40.085144 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:40.085151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:40.085687 (+ 536us) server_negotiation.cc:407] Connection header received
0807 11:35:40.085923 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:40.085929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:40.086046 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:40.086166 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:40.087122 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36430: BlockingRecv error: recv got EOF from 127.0.0.1:36430 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 11:35:40.127417 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:40.124149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.24.103.125:46365)
0807 11:35:40.124508 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:40.124514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:40.125609 (+ 1095us) server_negotiation.cc:407] Connection header received
0807 11:35:40.125835 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:40.125842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:40.125942 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:40.126084 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:40.127148 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36446: BlockingRecv error: recv got EOF from 127.0.0.1:36446 (error 108)
Metrics: {"server-negotiator.queue_time_us":185}
W20260807 11:35:40.198804 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:40.196749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.24.103.125:46365)
0807 11:35:40.196964 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:40.196972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:40.197322 (+ 350us) server_negotiation.cc:407] Connection header received
0807 11:35:40.197485 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:40.197492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:40.197619 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:40.197764 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:40.198554 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36448: BlockingRecv error: recv got EOF from 127.0.0.1:36448 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260807 11:35:40.332430 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:40.330188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.24.103.125:46365)
0807 11:35:40.330453 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:40.330459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:40.330806 (+ 347us) server_negotiation.cc:407] Connection header received
0807 11:35:40.331102 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:40.331107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:40.331188 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:40.331312 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:40.332141 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36464: BlockingRecv error: recv got EOF from 127.0.0.1:36464 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260807 11:35:40.594880 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 11:35:40.592889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.24.103.125:46365)
0807 11:35:40.593071 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:40.593075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:40.593461 (+ 386us) server_negotiation.cc:407] Connection header received
0807 11:35:40.593627 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:40.593631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:40.593710 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:40.593850 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 11:35:40.594656 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36470: BlockingRecv error: recv got EOF from 127.0.0.1:36470 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 11:35:40.776726 18944 catalog_manager.cc:1259] Loaded cluster ID: 656a28092b29432b8f032704e443d250
I20260807 11:35:40.776844 18944 catalog_manager.cc:1552] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285: loading cluster ID for follower catalog manager: success
I20260807 11:35:40.779635 18944 catalog_manager.cc:1574] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285: acquiring CA information for follower catalog manager: success
I20260807 11:35:40.781622 18944 catalog_manager.cc:1602] T 00000000000000000000000000000000 P be4ef70dbd6b4893bf7e43d279f3e285: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:35:40.874717 18958 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08fa8c7346224fccb899dd087eafbafd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 11:35:41.122627 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:41.111903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.24.103.125:46365)
0807 11:35:41.115568 (+ 3665us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:41.115578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:41.116170 (+ 592us) server_negotiation.cc:407] Connection header received
0807 11:35:41.116339 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:41.116345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:41.116552 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:41.116721 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:41.117219 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:41.118361 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:41.119883 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:41.120546 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:41.120672 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:41.121240 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:41.121931 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:41.122073 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:41.122209 (+ 136us) server_negotiation.cc:299] Negotiation successful
0807 11:35:41.122301 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3474,"thread_start_us":3312,"threads_started":1}
I20260807 11:35:41.136583 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:41.129645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.24.103.124:37799)
0807 11:35:41.130566 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:41.130579 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:41.130596 (+ 17us) server_negotiation.cc:407] Connection header received
0807 11:35:41.130691 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:41.130695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:41.130851 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:41.131077 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:41.131599 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:41.132507 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:41.133978 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:41.134496 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:41.134651 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:41.135259 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:41.135844 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:41.136053 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:41.136112 (+ 59us) server_negotiation.cc:299] Negotiation successful
0807 11:35:41.136203 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":538,"threads_started":1}
I20260807 11:35:41.150864 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:41.142984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.24.103.124:37799)
0807 11:35:41.143342 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:41.143351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:41.144485 (+ 1134us) server_negotiation.cc:407] Connection header received
0807 11:35:41.144756 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:41.144760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:41.144822 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:41.144870 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:41.145567 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:41.146636 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:41.148856 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:41.149502 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:41.149634 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:41.150124 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:41.150257 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:41.150353 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:41.150529 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 11:35:41.150613 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260807 11:35:41.152508 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:41.142780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.24.103.126:38461)
0807 11:35:41.143777 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:41.143786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:41.144236 (+ 450us) server_negotiation.cc:407] Connection header received
0807 11:35:41.144641 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:41.144650 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:41.144918 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:41.145099 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:41.146127 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:41.147626 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:41.149826 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:41.150415 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:41.150551 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:41.151243 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:41.151675 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:41.151768 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:41.152044 (+ 276us) server_negotiation.cc:299] Negotiation successful
0807 11:35:41.152210 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":673,"threads_started":1}
I20260807 11:35:41.154161 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 11:35:41.144469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.24.103.125:46365)
0807 11:35:41.144687 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 11:35:41.144695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 11:35:41.148202 (+ 3507us) server_negotiation.cc:407] Connection header received
0807 11:35:41.148370 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 11:35:41.148376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 11:35:41.148472 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 11:35:41.148572 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 11:35:41.149177 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:41.150047 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:41.152034 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 11:35:41.152703 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 11:35:41.152860 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 11:35:41.153435 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 11:35:41.153594 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 11:35:41.153697 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 11:35:41.153872 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 11:35:41.153954 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 11:35:41.158375 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 18642
I20260807 11:35:41.159680 18715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:35:41.276053 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 18642
I20260807 11:35:41.301940 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 18730
I20260807 11:35:41.303233 18798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:35:41.416770 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 18730
I20260807 11:35:41.442206 24989 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska8aRx7/build/asan/bin/kudu pid 18813
I20260807 11:35:41.443552 18880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 11:35:41.554878 24989 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska8aRx7/build/asan/bin/kudu with pid 18813
2026-08-07T11:35:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5338 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10488 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (388301 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST